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Preface 



For the thirty-second year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings . This is 
Volume #2 of the 3 1 st Annual Proceedings of Selected Papers On the Practice of Educational 
Communications and Technology. This volume includes papers presented at the national convention of the 
Association for Educational Communications and Technology held in Louisville, KY. Copies are available 
online at AECT.ORG. Volumes 1 and 2 are also available through the Educational Resources 
Clearinghouse (ERIC) system. 

This volume contains papers primarily dealing with instruction and training issues. Papers dealing with 
research and development are contained in the companion volume (Volume #1). 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings . The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 

Michael R. Simonson 
Editor 
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PowerPoint and the Pedagogy of Digital Media Technologies 

Catherine Adams 
University of Alberta, 

Edmonton, AB Canada 

With the ever-increasing importance of technologies as what orients us in the practical lifeworld, our 
extensive dependence on them has never been more central or more deserving of sustained critical attention. Indeed, 
since this dependence forms the very basis of our agency in the technologically mediated lifeworld, developing 
some understanding of and command over it forms the prerequisite for any subsequent practical project and must 
accordingly be considered the central concern of contemporary technocultural criticism. (Hansen, 2000, p. 258) 

The Internet, iPods, gaming systems and smart phones are changing the way we work, play and interact in 
the digital age. Similarly new media. Virtual Learning Environments, electronic whiteboards and new software tools 
are significantly altering processes of teaching and learning in primary, secondary and postsecondary education 
settings. Few are surprised that in virtually every classroom in schools, training institutions and universities, 
computers are commonplace. Students supplement textbooks by accessing their assignments and readings online, 
they wordprocess their course papers, download PowerPoint presentations and class notes, keep in touch via 
Learning Management System discussion boards and online social networks, all the while texting and twittering on 
their smart phones. New technological tools are changing how we learn, what we know, and how we understand and 
live in the world around us. Yet, we have barely begun to grasp the profoundly co-constitutive relationships we 
share with our digital technologies, relationships that simultaneously open new worlds of possibilities while silently 
foreclosing others. 

My research investigates this over-riding question: How are new media technologies, (re)shaping 
knowledge 1 , altering how it is represented, presented, and subsequently comprehended? The unique issue underlying 
this inquiry is captured in Marshall McLuhan’s notion of the invisible “lines of force” (1964, p. 15) that digital 
media technologies seem to be exerting in the educational context. To narrow the scope of my investigation 1 elected 
to study a now ubiquitous, relatively simple-to-use, software presentation tool: PowerPoint. Using PowerPoint as a 
touchstone, my research examines how software may extend but also serve to constrain what a student sees, 
experiences and has access to. In a similar manner, I investigate how teachers are not only aided and “enhanced” by 
PowerPoint, they are also enmeshed, constrained by and relinquished to the language, imagery, framing, at- 
handedness, and sensuality of its materiality and design. As Merleau-Ponty observes, “our existence changes with 
the appropriation of a fresh instrument” (1962, p. 143). We might wonder then, what transformations of perception 
occur, what translations of action manifest as teachers adopt a “fresh instrument” like PowerPoint in the lived space 
of the classroom? What is it like for students to learn via PowerPoint presentation? Does PowerPoint affect habits of 
mind? What is the nature of the vocative appeal digital technologies like PowerPoint seem to exercise in the lived 
space of the classroom? Can we catch glimpse of the new lifeworlds opened as teachers and students respond to the 
invitational quality 2 of these new media technologies? 

The PowerPoint literature so far 

Much of the educational literature on PowerPoint has focused on how-to advice and providing practical 
exemplars (e.g. Buchholz & Ullman, 2004). Some survey data suggest students have an overall positive attitude 
towards PowerPoint (Atkins-Sayre, Hopkins, Mohundro, & Sayre, 1998; Apperson, Laws and Scepansky, 2006; 
Daniels, 1999; Frey & Birnbaum, 2002; Harknett & Cobane, 1997; Kask, 2000; Lowry, 1999; Mantei, 2000; 
Nowaczyk, Santos, & Patton, 1998; Szabo & Hastings, 2000). Students report PowerPoint is a useful cognitive tool. 



1 My use of the term knowledge is intended to be inclusive of the passions, skills, attitudes, and emotions that inhere in teachers’ 
knowing. 

“ The “invitational quality” of a thing is very similar to J. J. Gibson’s original term “affordance” (now popularized in human- 
computer interaction and design literature, cf. Donald Norman). Gibson (1979) claimed affordance as “a radical hypothesis, for it 
implies that the ‘values’ and ‘meanings’ of things in the environment can be directly perceived” (p. 127). He credited his coinage 
of the term to Kurt Lewin’s description of the Aufforderunscharakter of environments and objects. Lewin (1926) illustrates: 

The beautiful weather, a certain landscape invites one to go for a walk. A staircase entices the two-year old child to 
climb up and jump down; doors entice one to open and shut them, little crumbs to pick them up, a dog to pet it; the 
sandbox to play in it; chocolate or a piece of cake to be eaten, etc. (p. 350) 

Around that time, American philosopher George Herbert Mead (1934) similarly wrote of armchairs “calling out” for us to sit in 
them (p. 278-80). Phenomenologically speaking, we often “hear” objects and aspects of the environment as invitations to partake 
of and participate in the world in particular ways. 
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especially when the electronic files or slide printouts are made available for review. They describe teachers using 
presentation software as generally more organized. On the other hand, a recent poll of 4,500 American 
undergraduates reveals significant student unhappiness with the way technology is being employed in lecture halls, 
most particularly PowerPoint (Kvavik, Caruso & Morgan, 2004; Young, 2004). 

Studies aimed at determining the efficacy of PowerPoint relative to other teaching methods have yielded 
mixed results. Lowry ( 1999), Mantei (2000), and Szabo & Hastings (2000) report PowerPoint-enhanced lectures 
increased levels of academic performance among college students, whereas Daniels (1999), Rankin & Hoaas (2001) 
report no effect. Kask (2000) found female, but not male, college students achieved better grades in a 
microeconomics course using PowerPoint. However, Susskind (2005 ) questions the results of some of these early 
studies, citing research design flaws. Apperson, Laws and Scepansky (2006), also in an attempt to overcome 
previous research design flaws, measured student satisfaction and test performance in ten classes across four 
disciplines. One semester was taught with PowerPoint, one without, with each pair given by the same professor. 
This study concludes PowerPoint does not impact academic achievement, but does develop an overall positive 
impression of the professor including likeability, organization, and a host of other “good” teaching behaviors not 
directly attributable to PowerPoint. Levasseur and Sawyer (2006), offering the most comprehensive review of the 
educational literature on PowerPoint to-date, similarly conclude that “the majority of studies comparing computer- 
generated slide -based instruction against other instructional methods have failed to find significant differences in 
learning outcomes” (p. 1 16). 

Critical analyses of PowerPoint have also been forwarded. Most notably, visual communications expert 
Edward Tufte (2003) claims PowerPoint supports a cognitive style that is inconsistent with both the development of 
higher analytical thinking skills and the acquisition of rich narrative and interpretive understanding. Some 
geography scholars, whose discipline is embedded in visual representation practices, worry PowerPoint is 
commanding an “epistemological monopoly [that] reinforces the interchangeability of content within the single 
(re)presentational system” (Crang, 2003, p. 239) and carries unfortunate corporate undertones (Matless, 2003; Rose, 
2004). In this way, PowerPoint may prove to be a “killer app” superceding a variety of classroom practices and 
potentially rendering obsolete valuable, perhaps critical, knowledge forms (Adams, 2006). 

Sherry Turkle (2004) suggests productivity software like PowerPoint “constitute a particular aesthetic in 
educational computing” (p. 101). PowerPoint promotes a particular way of thinking, one that “does not encourage 
students to make an argument [but rather] to make a point.” (p. 101). Digital media researcher Jamie O’Neil (2005) 
uses Bourriaud's theory of relational aesthetics to examine “how the medium of PowerPoint effects (or affects) the 
message” (p. 84). His intent is to dislodge the common instrumental, effective view of PowerPoint and install a 
critical, affective, experiential one. O'Neil concurs with artist David Byrne’s claim that PowerPoint “tells you how 
to think as it helps you accomplish your task” (Byrne, 2003, p. 3) and welcomes the arrival of “critical PowerPoint 
artworks (or covert interventions) as a mode of resistance to groupthink” (O'Neil, 2005, p. 84). 

What does all this mean for educators using PowerPoint in their classrooms? Studies show no significant 
gains in academic performance. At the same time, there is an appreciable increase in positive feelings towards 
instructors using PowerPoint. Finally, critical analyses are aligned on this point: PowerPoint tends to encourage a 
particular way of thinking, a way that may have questionable — or at least limited — merit in academic environments. 
Meanwhile, we still do not know how students or teachers actually experience PowerPoint mediated lessons and 
lectures. We are missing what Turkle (2004a) calls “the phenomenology of the digital experience” (p. 102). 

Methodology 

Mindful of Turkle’s suggestion that we may be missing a phenomenology of the digital experience, this is 
precisely the approach I take for my research methodology. The main focus and aim of phenomenological inquiry is 
the description of lived experience, that is, the description of phenomena as they present themselves or as they are 
given in experience. Phenomenology is concerned with how we experience our world pre-reflectively , pre-verbally 
in its lived immediacy; it is the practice of fidelity to lived experience. As well as describing experience, 
hermeneutic phenomenology seeks to draw out the meaning or significance of our practical involvements in the 
world. Such research formulates questions of the type, “What is this or that human experience like?” It is an attempt 
to return “to the things themselves” (Husserl, 1911/80, p. 1 16), and further, to let these things (phenomena) speak 
for themselves (Heidegger, 1962). Phenomenology is not interested in conceptualizing, theorizing or idealizing 
experience, but rather in describing and interpreting experience as it is lived. 
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Phenomenological inquiry explicitly positions the researcher to comprehend information and 
communication technologies, not as solely objective or subjective phenomena, but as lived. A central feature of 
phenomenological method is the gathering of a field of descriptive evidence from which underlying patterns and 
structures of experience can be drawn (van Manen, 1997). My study addresses three distinct modes of PowerPoint 
engagement: (1) how PowerPoint presentation is experienced by students; (2) how teachers experience constructing 
a presentation with PowerPoint; and ( 3) how teachers experience teaching through PowerPoint presentation. 

The phenomenological study involved in-depth interviews with fourteen college students and twelve 
instructors at two different academic, post-secondary institutions; observation of large university lecture classes 
where PowerPoint was being employed as a primary means of teaching; and reflection on my own use of 
PowerPoint as an instructor in post-secondary settings. I used hermeneutic phenomenological methodology to 
capture the particularities of the PowerPoint experience in the form of lived experience descriptions (LEDs). 
Methods employed include thematic analysis, linguistic interpretations, honing of exemplary or anecdotal narratives 
through eidetic reduction. Phenomenological method also requires a systematic scholarly “reading” of relevant 
philosophical literature and phenomenological studies. Using techniques such as comparing pedagogical styles of 
classroom discussions and presentations with and without PowerPoint I examine how the experiences of software- 
mediated presentations are uniquely sponsoring and providing for modes of teaching and learning that are always 
and inevitably embodied and situated in particular temporal, spatial and relational contexts. 

Theoretical Framework 

The totality of the immediate environment that we inhabit, our lifeworld. is best described as “a milieu — a 
field of intensive forces, vibrant according to their own inner codes” (Lingis, 2004, p. 278). Ivan Illich (1996) coins 
the phrase le milieu technique to refer to the irresistible embrace of the high technology lifeworlds we find ourselves 
dwelling in today. The technological milieu is shaping substantially — insinuating itself, habituating us and 
simultaneously reinterpreting — how we act in and perceive the world. Mark Hansen suggests that new media 
technologies are “poised on the cusp between phenomenology and materiality” and as such have introduced “a 
theoretical oscillation that promises to displace the empirical-transcendental divide” (2006, p. 297) that has long 
structured western thinking. 

This research is situated in the midst of this difficult theoretical divide, and attempts to make visible some 
of the tight intimacies, primordial interminglings, and, at times, acute dependencies teachers and students find 
themselves living with their educational technologies every day. To this end, this research project draws on lour 
phenomenologically-informed traditions: curriculum and pedagogy studies (e.g. Max van Manen, Iain Thomson), 
philosophy of technology (e.g. Martin Heidegger, Don Ihde), human environmental aesthetics (e.g. Berleant & 
Carlson, Pauline von Bonsdorff) and media studies (e.g. Hansen, Marshall McLuhan). For example, in the section 
below, I frame participants’ lived experience descriptions in light of Martin Heidegger’s foundational insights about 
human-technology relations. Heidegger, one of the earliest philosophers of technology, shows that each thing (or 
place) opens a new world to us, revealing novel structures of experience and meaning; every technology discloses a 
new horizon of possibilities to us. Human beings are “the be-thinged” (Heidegger, 1971, p. 181), that is, we are 
prereflectively inhabited, conditioned, and creatively provoked by the things of our world. 

In the remaining pages, I provide an excerpt of the final phase of my project, which attempts a 
phenomenology of university teachers’ everyday experiences of PowerPoint. 

Inhabiting PowerPoint: constructing a lesson with PowerPoint 

The vocative appeal of PowerPoint to the teacher is at once a linguistic gesture: “Click to add title,” “• 

Click to add text”. It is also a promisingly familiar and easy-to-use digital environment; a hermeneutic horizon of 
previous PowerPoint design and teaching experiences; entrance to an architected form intending persuasive 
presentation; a windowed milieu that the teacher traverses with her eyes upon screen, fingertips on keyboard, hand 
shuffling a mouse. As Heidegger (1972) tells us, “When we handle a thing, for example, our hand must fit itself to 
the thing. Use implies a fitting response” (p. 187). Reaching out with anticipation of PowerPoint' s promise to help 
her point powerfully, the teacher orients herself toward her windowed environment; her being is drawn in and gently 
caught up in the “draft” of PowerPoint, the unique horizon of possibilities it brightly frames for her. She responds 
fittingly. 
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One teacher describes how she constructs a lesson using PowerPoint: 

I insert an image, add some text, then try them in different positions on the slide. I’m looking for balance. 1 
like using compelling images, with minimal, carefully chosen text for impact. A.? I work, I do not, cannot 
separate the composition of slides themselves from the subject matter at hand, the vision of my students, 
and the appeal I am trying to make. I sit back and look (perhaps trying to see the slide as my students 
might), then adjust, and adjust things again. I try out different fonts, sample background colors from my 
images, wanting to give the whole presentation a sense of visual cohesion. I take a certain pleasure and 
satisfaction in this. I move to Slide Sorter View [where all the slide thumbprints are laid out across the 
window] to grasp the whole so far, to visualize the general flow of the presentation. From here, I move a 
few slides to a different place in the sequence to see how that flows, then return to Normal view. I find I am 
variously engaged with trying to represent the content, the purpose of this teaching presentation, visually, 
in text, or both, and thinking about, imagining presenting the slides to my class. 3 

Within the PowerPoint environment or milieu, the teacher’s work materializes as an accumulating series of slides. 
The basic elements of each slide are text, images, color, and animation. She composes, adjusts, tries out new fonts, 
samples colors, switches “views,” plays with order. She is wholly engaged, representing content as slides then 
imagining their presentation in the immediacy of a classroom with her students. Slides, subject matter, the vision of 
her students, and her presentational and teacherly intentions intermingle. 

In performing this preparatory work, the teacher is sitting in her office with computer, screen, keyboard and 
mouse; texts and papers litter the desk. Her screen shows numerous windows open: a web browser, email, a Word 
document, as well as PowerPoint. Occasionally her eyes wander from the screen, and stare thoughtfully out her 
office window into the distance. She turns back to the PowerPoint window, pulls her keyboard a little closer, nudges 
her mouse and continues work. Once the teacher is engaged in her preparation work, her office, desk, screen, 
keyboard and mouse recede into the background. PowerPoint too withdraws from full view, fading to a transparent 
framework, a sophisticated but peripherally present set of tools that she may variously call upon to perform her 
presentation design activities in this digital world. 

Phenomenologically speaking, we do not usually engage a tool as a discrete, obvious object, that is, in what 
Heidegger calls its present-at-hand mode ( vorhandenheit ). Rather we tend to encounter a tool through using it, in its 
handiness ( zuhandenheit ). In this handy encounter, the tool is essentially invisible to us, taken-for-granted. It is, as 
Sartre says about the everyday experience of our own bodies, passed over in silence — “ passe sous silence ” (Sartre in 
Bleeker & Mulderij, 2002). 

Consider the example (used by Wittgenstein, Polyani, and Merleau-Ponty) of the blind man’s cane. We 
hand the blind man a cane and ask him to tell us what properties it has. After hefting and feeling it, he tells 
us that it is light, smooth, about three feet long, and so on; it is occurrent for him. But when the man starts 
to manipulate the cane, he loses his awareness of the cane itself; he is aware only of the curb (or whatever 
object the cane touches); or, if all is going well, he is not even aware of that. . ..Precisely when it is most 
genuinely appropriated equipment becomes transparent. (Dreyfus, 1991, p. 65) 

Hammering a nail in the wall to hang a picture, we are focused on the picture-hanging, the project we are engaged 
in, not on the hammer. When we are writing a scholarly paper or an email, we are barely aware of our typing fingers 
or the keyboard. Our fingers serve us silently, tapping transparently on the vaguely present keyboard, while we are 
primarily engaged in the higher-level business at hand: writing thoughtfully. Only when we accidentally hit our 
finger with the hammer do we suddenly awaken to our throbbing finger and to the hammer as an obvious object. 
Too, until our fingers know how to type, the keyboard stands as an insurmountable obstacle. But, as Heidegger 
(1962) describes, “the less we just stare at the [tool], and the more we seize hold of it and use it, the more primordial 
does our relationship to it become” (p. 98). To be what it is, a tool must recede from visibility. 

The work-object or focal project of our instructor is thus not PowerPoint. Her project is the classroom 
situation she will find herself in a few days hence. As teacher, her primary intention is to creatively assist her 
students in learning the particular subject matter at-hand. For this purpose, for this subject matter, she has chosen to 
use PowerPoint. Thus while the presentation software expertly frames and facilitates her activity of planning a 
lesson, PowerPoint is not the main objective and intention, anymore than canvas and paint palette are the objective 
and intention of the artist. Nonetheless, we must also notice how the instructor’s activity patterns and meaning 
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The italicized text represents phenomenological research material drawn from interviews conducted with twelve university and 
college instructors regarding their everyday, “lived” experiences of PowerPoint. Individual participants are identified only as “the 
teacher”, “the instructor”, “he” or “she”. 
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structures are also being quietly in-formed — conformed, deformed, and reformed — by the architecture of the 
particular software she finds herself inhabiting. 

The PowerPoint habit 

Another teacher relates simply: “ Sitting down to prepare a PowerPoint, I can ’t help but think in bullets. ” 
Parker (2001) humorously notes how PowerPoint seems to promote a certain kind of thinking: “Last week I caught 
myself planning out (in my head) the slides I would need to explain to my wife why we couldn't afford a vacation 
this year” (p. 78). As the teacher seizes hold of PowerPoint as a tool of teaching, he or she necessarily begins to 
think in terms of the form it suggests. At minimum the teacher must think in slides, reconfiguring his or her 
knowledge in the new 4:3 rectangular landscape delineated by PowerPoint. The software readily assists in this 
project by inviting the teacher to consider certain formats: to title each slide, to reform subject material as 
abbreviated, bulleted points. 

In PowerPoint, the teacher sees and understands her teaching world in terms of the sphere of possibilities 
this software discloses to her as she works: slides, menus, animations, Slide Sorter View, Normal View. Her lesson 
planning world unfolds in the context of a bright, spacious rectangular “window,” a bright screened facade framing 
and containing explicit text and iconic invitations. Ihde (1990) suggests, “technologies, by providing a framework 
for action,. . .form intentionalities and inclinations within which use-patterns take dominant shape” (p. 141). In 
PowerPoint, the teacher “does not, cannot separate” the software’s possibilities and designs from her own: the aims 
and inscriptions of the Microsoft programming team and the teacher intentionalities and inclinations intertwine, 
enmesh and reorient. The teacher’s world is translated into new vocabularies and presentation genres, expanding her 
possibilities of action while simultaneously framing and constraining the world as a screenic succession of 4:3 
slides. 

Having responded to the vocative appeals of PowerPoint — its invitational qualities or affordances — the 
teacher enters a mode of human-technology engagement Chesher (in Suchman, 2007) describes as “managed 
indeterminacy” or invocation. “Invocation involves those actions that define the terms of engagement written into 
the design script or discovered by the participating user” (Suchman, 2007, p. 282). The teacher is now 
conversationally engaged, enfolded into, and intertwined with PowerPoint. The teacher-technology relational 
boundaries blur and a hermeneutically rich but “silent” corporeal rapport sets in. What sorts of conversations seem to 
unfold between teacher and PowerPoint? 

Yet another instructor describes a somewhat different approach and concern when creating her PowerPoint 

slides: 

Composing this slide, there was a particular aesthetic I was striving for: thoughtful use of color, thematic 

cohesiveness, consistency between the slides (not sameness!), but also movement, meaningful movement 

through and among the slides. There is clearly an art to this. 

This teacher is preoccupied with visual appeal and attaining thematic integrity with the subject matter. Movement 
achieved “through and among” her slides has significance to her. The teacher is trying to be sensitive to the 
atmospheric quality of the PowerPoint media on her students. This raises the question of how atmosphere is usually 
anticipated in the planning of a lesson and how the aesthetic of PowerPoint slides may be seen as an evocative tool 
for establishing a sphere. 

“ PowerPointing ” : Teaching by and with PowerPoint 

Enter teacher with trolley replete with laptop, mouse and data projector. Untangling the garage-band knot 
of electrical cords and connector cables, the teacher connects, plugs in, and turns on laptop and projector. This 
process is sometimes accompanied by palpable anxiety surrounding the stages of equipment hook-up, and worries 
about self-competence in the face of difficulties or breakdown and the implications of “no PowerPoint” to the fate of 
the class. The projector hums at last, the slides are cued up, the teacher breathes a quiet sigh of relief. 

The simple act of drawing the blinds or switching off the light, darkens perceptibly the hue of the wall, 
softens the faces of students. The teacher becomes less visible; the projected slide shines brighter. The mood 
changes, the classroom atmosphere shifts. PowerPoint reconfigures the classroom as a cinematic space: the students 
settle in as spectators, the teacher as orator, narrates the slides from the side. When the teacher turns to the opening 
slide, the students are cued to sit back, get comfortable and hopefully enjoy and learn from the PowerPoint 
presentation with a certain sense of (inter)passivity. A subtle change occurs in the students’ attitude and orientation. 
The large, bright slideshow reminds students they may become a particular kind of audience, “invigorated or 
drowsy, [but] a generally passive audience that is rarely called upon to really interrogate the images” (Crang, 2003, 
p. 242). But what does the projected PowerPoint slide evoke for the teacher? 
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The vocal rhythm of PowerPoint 

I notice when I turn to begin my PowerPoint, I shift my role slightly — Pm less conversational, more 
oratorical. PowerPoint locks you into a gait in your speech, a kind of vocal rhythm. 

The teacher with-PowerPoint finds himself standing somewhat differently in relationship to his class: less dialogic, 
more monologic; less open to interruption and discussion, fastening to a vocal pattern that rhythmically signals 
oration not conversation. Vocal rhythm may also establish a kind of synchrony with the slide rhythm. 

The arrival of a new slide is the occasion to take a breath, a momentary pause to look at the slide, allow its 
meaning to prompt me: a reminder of what to say next, what direction to pursue. But too, I must somehow 
find connection with what I have just said. Or not. It tells me what comes next. I feel I must press on. 

Like walking and talking with a good friend, footfalls — narrative breaths and slidefalls — find a mutually 
comfortable rhythm and pace. Here a special kind of pathic relation is hosted, not between teacher and students, but 
between teacher and projected slides. This human-technology dialogue appears more determined and rigid than the 
comparably flexible, nuanced relation engaged during the planning and design phase. More specifically, the slides 
are no longer in the midst of being created and manipulated. In View Show” mode, the teacher cannot change the 
slides themselves, he can only control the direction of movement between the slides and animation moments — 
forward, backward — as well as access preset links and buttons. This predicament of being instructionally captivated 
in a slide set seems too to be a consequence of the teacher planning the lesson with a series of headlines or points, as 
we saw above. 

In the lived moment of the class, as each new slide appears, it “speaks” prereflectively to both students and 
teachers alike. That is, before we are even aware of it, we have already seen the new slide and begun to read and 
make sense of it; the teacher finds himself or herself speaking to the slide. When the next slide is summoned, it 
“speaks” again, and the teacher must now speak to it. 

“Iam committed to do this PowerPoint" 

As soon as I clicked to the next slide, I knew immediately it was the wrong thing. Seeing their eyes, I felt: I 
simply can ’t go on. It was the same sinking feeling you get realizing the person you are having a 
conversation with isn ’t listening to you. I had spent all this time preparing this PowerPoint presentation 
and then the problem with PowerPoint is you just can ’t simply jump ahead, be extemporaneous — “just 
ignore this and this while I find the right slide. ” I was stuck with my plan. 

This college instructor recalls a time when he suddenly felt that, in the lived context of his class, his choice of using 
PowerPoint to address a particular topic was misjudged. Of course, any lesson plan or teaching approach can go 
awry or fall flat. In such moments, the teacher may decide to “stick with the plan" or diverge and improvise. The 
seasoned teacher usually has a few other “tricks” at-hand. Yet, is there something about PowerPoint that complicates 
the move to diverge in response to one’s felt sensibilities? One teacher describes her PowerPoint dilemma like this: 
PowerPoint is a finished product. It is hard for me to loose myself from the slides in the context of my class. 
The story has, so to speak, already been decided. 

But perhaps, the problem is precisely that the story had not been decided. The teacher did not prepare a story but a 
series of points, stops on the way to some cognitive end point. She goes on to describe the resistance she feels in 
deviating from the slide set she herself has constructed: “If I answer a question, how will I go back to the slides?” In 
planning and carefully constructing the lecture beforehand, she tried to imagine her students there before her, tried to 
anticipate their questions. But now, in the context of her actual class, the world looks different. 

In the classroom, PowerPoint is a representation of my anticipated presentation — an imagining of what my 
presentation would be, could be. But in the actual moment of teaching, things are often otherwise. In the 
midst of teaching, my slides and I sometimes come into conflict with one another. Then I feel fragmented, 
forced to choose this particular outcome — what is represented up there on the slides — over the felt relation 
with my students — what seems to present itself to me in the moment. Iam committed to do this PowerPoint. 
I cannot now easily choose to do something else. 

When a teacher uses PowerPoint in her classroom, she commits to the unfolding of a particular form of teaching and 
learning, a predetermined story wending its reckoned path to a decided conclusion. A PowerPoint presentation 
prepared beforehand is also an investment, visible proof of preparation and organization in the face of the 
contingent, indeterminate lifeworld of the classroom. To abandon such obvious evidence of competence may strike 
as fool-hearty, exposing oneself to an uncertain, unprepared-for future. As Howell (2007) laments 

From the moment I walk into the lecture theatre I feel the pressure from my students to line up my thinking 
with their PowerPoint notes, without which they seem to be lost. I usually succumb by connecting them to 
the screen rather than to myself, each other, and the subject matter. In giving precedence to the object of 
PowerPoint, where the slides take on a language and world of their own,... students may subconsciously be 
encouraged to zoom out of the teacher’s presence in favor of the rectangle on the screen, (p. 139) 
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The Times-Square-like surround of slick and easy possibilities is so appealing and omnipresent, our inner compass 
as teachers may be quietly lifted from us and replaced by the veneer of “powerful” solutions. As sociologist Daniel 
Bell prophetically wrote in the early 1970s, the new “intellectual technologies” — tools that specifically extend our 
cognitive reach — substitute “algorithms (problem-solving rules) for intuitive judgments” (1973, p. 29). A digital 
technology is given default proxy for professional knowing. 

Ready, set, teach! 

Not so long ago, I gave a lecture for a PowerPoint-loving colleague of mine who had to be away. Standing 
before his students, I opened his PowerPoint file on my laptop, the whole system struggling to cope with the 
gigantic file. While we are waiting, I tell his students that their professor has left me 143 slides to cover 
today. “That means, ” I calculate, “one slide every 21 seconds. So we better hurry up and get started!” 
PowerPoint exhibits the tendency toward or certainly the desire to achieve maximum efficiency in teaching. 
Contemporary technologies are both the product of as well as the increasingly complex scaffold supporting and 
reifying a particular technological frame of mind, “a mode of revealing,” which Heidegger calls “enframing” (das 
Gestell). In today’s ubiquitous surround of technologies 

we increasingly think and act in accordance with the world picture [modern technology] provides. . .The 
technological mode of revealing is a fixation of things by categorizing them and representing them to 
ourselves in thought through abstract categories, thus making manageable and capable of being efficiently 
manipulated — a demand to which the fluid and the ill-defined remains inconveniently resistant. . ..We 
“enframe” things by turning them into instances — understanding them in terms of the objective properties 
attributed to members of the category to which they have been allocated. (Bonnett, 2002, p. 234). 

This technological way of apprehending things — wherein all things, including human beings, increasingly show up 
to us as resources to be enhanced and optimized for maximal efficiency — is radically restructuring our daily lives, 
along with contemporary learning experiences and teaching practices. To put it another way and perhaps a little 
more forcefully, post-modern technology engenders a totalizing style of practices that, according to Dreyfus and 
Spinosa (2003) threaten to: “restrict our openness to people and things by driving out all other styles of practice that 
enable us to be receptive to reality. This threat is not a problem for which we must find a solution but an ontological 
condition that requires a transformation of our understanding of being. For that, we need to understand technicity as 
our current mode of revealing things and people” (p. 341). 

The demand to “have" the PowerPoint 

On the first day of class, a student asks, “Will you be making your PowerPoint slides available?” I reply, 
“I haven ’t yet read all of Plato ’s dialogues, nor have I learned yet how to put slides up on the web. Given a 
choice between taking the time to read another dialogue and putting my PowerPoint files on the web, I 
think Pd choose the former. ” At the back of the lecture hall, a young woman snaps her book shut, gathers 
her things and promptly leaves my classroom. 

The PowerPoint slide deck is a lecture product that students are increasingly expecting to procure from their teacher. 
In becoming a product, the teacher’ s work may seem less a matter of developing pedagogic relations and the sharing 
of understanding, skill and expertise, and more a matter of commodity and consumption. Here the young woman 
expresses her disgruntlement that the new covenant of entitled student-consumer has been broken. She has nothing 
to gain from the philosopher in his person, only his PowerPoint. Another example: 

At a conference recently, where PowerPoint is the norm, lam speaking before a fairly large group. As I 
begin, I am surprised to notice someone, severed rows back, raise their hand as if for a question. But then I 
see the hand is holding a camera, and it quickly goes back down again. Next slide. The same digital- 
camera-hand goes up then down, and now, off to my right, some ways back, I see another camera-touting 
hand shoot up. I feel taken aback. Surely my PowerPoint slides are not so compelling that each slide 
should warrant photographing. No: I, or rather, my work, is being consumed, commoditized and 
owned... and all without my consent. 

Borgmann (1984) claims modern technology is decisively separating means from ends. The activities or processes 
of creating things are progressively being hidden from view and replaced with the more singular activity of 
procuring end-products or commodities. “What distinguishes a [modem] device is its sharp internal division into a 
machinery and a commodity procured by that machinery” (p. 33). As a result, some of the practices associated 
traditionally with creative teaching activities are ostensibly disappearing in the wake of sophisticated technologies. 

As illustrated above, the PowerPoint slide deck is essentially a product of a teacher’s knowing and thinking in 
conversation with the PowerPoint software, now inscribed as single framed, sequential snapshots. Thus with PowerPoint, 
students witness more often the projected knowledge product, and less the teacher’ s knowing-in-action. Then again, each 
slide has the potential to trigger the embodied insights of an experienced practitioner in the immediacy of the now. This 
punctum or evocative capacity can “save” a PowerPoint presentation from being merely a product. 
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Yet it may be that “the ultimate success of teaching actually may rely importantly on the “knowledge” 
forms that inhere in practical actions, in an embodied thoughtfulness, and in the personal space, mood and relational 
atmosphere in which teachers find themselves with their students” (van Manen, 1995, p. 48). Thus, a primary 
concern here is a bypassing of the experiential dimensions of practical knowledge, both in the discipline of the 
subject as well as in teaching practice. When educators try to capture and translate aspects of their tacit 
understandings to a series of slides, there is the danger of “short-circuiting” the normally contingent enactments of 
their ordinary teaching and professional actions. Of course, “shortening the circuit” is precisely what devices of 
expedience, like PowerPoint, are designed to do: eliminate “unnecessary” sub-steps (via hardware or software 
solutions) to allow the most efficient path to an end. 

Overview of Findings 

Our corporeal being — our lived body — is increasingly and intimately enhanced by, enmeshed with and 
enfolded into new digital technologies. These paratextual 4 machines mediate our lived experience with startling 
immediacy and complexity, lending us novel sensory worlds, and pre-scribed ways of knowing and doing that are 
increasingly shared globally. The moniker “digital” is signaling a radical change in our material world, but also in 
our human selves. Techno-utopian thinkers like Hans Moravec and Ray Kurzweil predict human-technology fusions 
where the “software” of our minds will one day be uploadable to more durable, faster hardware, thus rendering our 
“mere jelly” (Moravec, 1988, p. 117) bodies — the “old slow carbon-based neural-computing machinery” (Kurzweil, 
1999, p. 129) — obsolete. In the wake of such euphoric claims of transcendence, philosopher N. Katherine Hayles 
(1999) reminds us that the “human mind without human body is not human mind. More to the point, it doesn’t exist” 
(p. 246). Our human self is intimately tethered to the possibilities as well as the limits of our flesh-and-blood, human 
body. Thus, as the “mere jelly” body is gradually being relinquished in these technology turf wars, a new version of 
human being has been conceived: the post-human, “whose basic capabilities so radically exceed those of present 
humans as to be no longer human by our current standards” (Bostrom, 2003, p. 5). 

What does all this mean for educators employing new media technologies? Foremost, we must begin to 
discern and “focus on our own embodiment as the material site — the bearer — of technology’s otherwise wholly 
inhuman impact” (Hansen, 2000, p. 263). Digital technologies are locally deployed “mimetic vehicles” (Benjamin, 
1978) that prereflectively shape our embodied agency. “Software quite literally conditions existence” (Thrift, 2005, 
p. 241) by scaffolding a habituation process that occurs primarily outside of the phenomenal field of subjectivity. 

One of the difficulties in grasping the mediating influence of software is that its texts do not fit the usual model of 
representation, wherein humans and objects represent each other via words and images. Instead, software texts 
concern words doing things in particular contexts: the language of the machine has immediate material effects. 

Our interactions with software, often via a screen and keyboard/mouse/controller, are direct, sensuous and 
mimetic. Software “affects our experience first and foremost through its infrastructural role, its import occurs prior 
to and independently of our production of representations” (Hansen, 2000, p. 4). PowerPoint sponsors a style of 
thinking and presenting, a normative framework for staging knowledge: headings and bullet points for teachers to 
“talk to”. This scaffolding of abbreviation, built into the software as default signage, implicitly informs how some 
teachers visualize and subsequently present their knowledge in the lived space of the classroom. The projected 
PowerPoint slide presentation, regardless of the kind of knowledge it is serving to frame, exercises a powerful sway 
over the teacher in the moments of teaching, at times appearing as impenetrable obstacle, rather than a generative 
support to the teacher desiring to pursue her pedagogical sense of tact. In this way, our lived experience is being 
radically, prereflectively re -habilitated; our intentional involvements perturbed and re-inscribed via the constraints 
and dispensations of pre-fabricated digital architectures. We are now well into an era of technological-becoming, our 
sensible bodies quietly adapting to the inhuman rhythms of an evolving, digitally coded and intensifying 
mechanosphere. 

It is imperative that we attend mindfully to the material, hermeneutic, and existential shifts that are 
transpiring as our worlds are daily extended, intensified, and complicated by digital technologies. The continued 
promotion of digital technologies as neutral agents — a foundational belief or “posit” of our current ontological 
epoch — imperils the normative project of pedagogy by concealing the instrumental constructs they materialize. 
Rather, these paratextual machines must be recognized as effective and affective mimetic interventions that 
prereflectively shape our being, knowing and doing in the world. Such a view necessarily burdens tomorrow’s 
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The term “paratextual” is used by Gerard Genette in his book Palimpsestes (1982) to describe "accompanying productions” 
that bind the text and the reader together. He lists the following as examples of paratexts: “title, subtitle, intertitles; prefaces, 
postfaces, notices, forewords, etc; marginal, infrapaginal, terminal notes; epigraphs; illustrations; blurbs, book covers, dust 
jackets, and many other kinds of secondary signals.” 




teachers with a renewed sense of professional responsibility, one sensitive to the uncertain ecology of today’s 
classrooms in the wake of digital technology integration, but more importantly, for the well-being of our post-human 
children living this brave new world. 

Finally, this research provides argument, ground, and example of how educational scholars must 
qualitatively document and critically theorize media technology’s “hidden” curriculum, that is, the material effects, 
activity patterns and meaning structures ICTs are mobilizing at a startling speed in education. Educational 
technologists have spent much energy in trying to assist teachers to use these new media tools, but through this 
research I show how we must also concern ourselves with how these tools may be using us. 
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Signaling the Keywords or Sentences: An Eye-tracking Study 

Ismahan Arslan- Ari, Kursat Cagiltay 



Abstract 

This study examined whether signaling the relevant keywords or signaling the relevant sentences was more 
effective on students’ performance and what the effects of signaling on reading with the help of eye-movements are. 
28 undergraduate students voluntarily participated in the study. 14 students studied the computer-based material in 
which relevant sentences were signaled and others studied the computer-based material in which relevant keywords 
were signaled. The results showed that type of signaling does not make any change on the students’ performance in 
retention and transfer scores. Eye-tracking data showed that signaling enhanced the attention, decreased the 
unnecessary visual search and directed the attention to the relevant information. Hence, it reduced the extraneous 
cognitive load because of split attention. 

Introduction 

Advances in the technology during recent years have increased the use of multimedia learning. Multimedia 
learning environments present the information in textual, graphical and audible forms. According to multimedia 
principle, “people learn better from words and pictures than from words alone” (p.l 17) (Fletcher & Tobias, 2005). 
Mayer and Morena (1999) stated that computer-based multimedia learning environments including pictures and 
words provide “a potentially powerful venue for improving student understanding.” (p. 108). However, only using 
the pictures with words is not enough to enhance the learning. “The case for multimedia learning is based on the 
idea that instructional messages should be designed in the light of how the human mind works” (Mayer, 2001; p.4). 
For this reason, instructional designers should be sensitive to cognitive theory of multimedia learning while 
designing the multimedia instruction. 

Cognitive theory of multimedia learning is based on engaging the active cognitive processing for 
meaningful learning (Ausebel, 1968; Wittrock, 1990 as cited in Mautone & Mayer, 2001). Cognitive theory of 
multimedia learning has three cognitive science of learning assumptions. People have separate channels for 
processing visual information (e.g. on-screen text, pictures and graphs) and auditory information (e.g. narration and 
background sound). The second one is the limited capacity assumption. According to this assumption, people have 
limited capacity to process information in each channel at one time (Mayer, 2005). This means that people have 
limited working capacity, whereas the capacity to store knowledge in long-term memory is unlimited. Sweller 
(2005) stated that instructional designs ignoring the working memory limitation “are likely to be random in their 
effectiveness”. The last assumption is that humans actively engaged in three cognitive processes which are selecting 
relevant information, organizing this information logically and integrating it with other knowledge (Mautone & 
Mayer, 2001). 

Improper instructional designs, which ignore the human’s limited working capacity, result in extraneous 
cognitive load (Sweller, 2005). Multiple sources of information such as texts, figures or narration are used in 
multimedia learning environments at the same time. However, processing all of this information simultaneously in 
working memory results in cognitive load (Sweller & Chadler, 1994). Mayer and Moreno (2001) stated that 
cognitive load is crucial in multimedia learning environments. Thus, they suggested nine ways to reduce cognitive 
load in multimedia learning. These solutions are off-loading, segmenting, pretraining, signaling, weeding, aligning 
words and pictures, eliminating redundancy and synchronizing. 

Using both words and figures in multimedia instruction sometimes results in unnecessarily splitting the 
learners’ attention. Learners should be kept information in the text in working memory while searching the referents 
in the figure. Ayres and Sweller (2005) stated that forcing the learning to engaging in a search in order to locate 
relevant information, then cognitive sources may be distracted from learning. For this reason, search process 
increase extraneous cognitive load. Proposed solutions should be based on decreasing the unnecessary search. 
Integrating texts and diagrams are one of the solutions to overcome this problem (Ayres & Sweller). Evidences have 
showed that learning is enhanced when related text and pictures are in close proximity in an integrated format rather 
than separated in space (Moreno & Mayer, 1999). Another proposed solution use using color coding to reduce 
cognitive load. Kalyuga, Chadler, and Sweller (1999) demonstrated that the learners who studied the color-coded 
format had lower working memory load and outperformed on a multiple choice test than the learners who studied 
the conventional format. The reason is reducing the unnecessary visual search. 

Because of the unnecessary visual search to find relevant information in the diagram, some processing 
resources in the mind will be consumed (Kalyuga et ah, 1999). For this reason, there will not be enough processing 
resources in the mind (Sweller, van Merrienboer, & Paas, 1998). As a result of the cognitive load associated with 
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the presentation format of instruction, performance will diminish. To minimize this problem, Mautone and Mayer 
(2001 suggest signaling, providing cues the people with the efficient and effective way to process the materials. 

In this study, we used signaling strategy to reduce the cognitive load caused by the combination of graph 
and explanatory text. Signaling guide the learner’s cognitive processing and help the learner to select the relevant 
information, organize the information logically and integrates the information with prior knowledge by providing 
cues (Mautone & Mayer, 2001). Using typographical cues, such as; underlining, capitalization, italics, bold-face and 
color variations, can be used for signaling (Lorch, Lorch & Klusewitz, 1995). These typographical cues can be used 
for taking attention to key terms, introducing technical terms, and emphasizing the important information. Lorch 
(1989) stated that “the simplest way to signal the relevance of information is to distinguish it visually from the body 
of the text by typographical variation” (p.223).Moreover, headings, titles, enumeration signals (first, second, etc.), 
arrows, linguistic cues (e.g. lower intonation) and outlines are the other cues for cognitive guide (Mautone & Mayer, 
2001 ). 

Previous research studies shows that signaling has an effect on students’ learning (Mautone & Mayer, 

2001; Lorch et al., 1995). Thus, people learn more deeply when the text is signaled. Mayer (1975) stated that “when 
the material is presented as text, signaling may include the use of highlighting of words. . .” (as cited in Mautone & 
Mayer, 2001). However, Lorch et al. (1995) signaled the target sentences instead of words in their study. Thus, it is 
not clear which one is more effective. 

Mayer (2005) stressed the need of extra studies to determine the signaling effects in a multimedia 
environment, because previous research studies are based on printed text. Moreover, in this study, eye-tracking 
data provided fixation duration and fixation count. These are valuable information to learn the cognitive processes of 
the learner during learning. 

Purpose of the Study 

This experimental study investigated the signaling effect in multimedia learning environment. In this study, 
two types of signaling strategy was used to reduce the learners’ cognitive load. One of them was signaling the 
relevant keywords, other one was signaling the relevant sentences. In the literature, there is no consensus about 
signaling word or signaling sentences in multimedia learning environments. This study aimed to give a clear 
understanding about whether signaling the relevant keywords or signaling the relevant sentences was more effective. 
Thus, one of the goals of the experiment was to examine the following questions. 

• Do the retention test scores for the participants who study with the multimedia learning material in which 
relevant keyword are signaled significantly differ from the scores for the participants who study with the 
multimedia learning material in which relevant sentences are signaled? 

• Do the transfer test scores for the participants who study with the multimedia learning material in which 
relevant keyword are signaled significantly differ from the scores for the participants who study with the 
multimedia learning material in which relevant sentences are signaled? 

Another goal was examining the role of signaling in multimedia learning environments by the help of 
fixation duration and learners’ locus of fixation while studying the material. This study shed on light how signaling 
affects the learners’ reading. 

Real-time measures of cognitive processing during multimedia learnin can be obtained from eye movement 
data such as number of fications, average fixation duration and total gaze duration (Henderson, Brockmole, 
Castelhano, & Mack, 2007). According to the eye-mind hypothesis , the location of the eye fixation is associated 
with what the subject is processing at that time and the duration of the fixation shows how long it takes to process 
that information (Just & Carpenter, 1976). 

Eye-tracking data is also beneficial to determine the efficiency of the instructional material in terms of how 
easy it is to find desired information (Murata & Furukawa, 2005). Number of fixations is smaller when it is easy to 
locate necessary information (Greene & Rayner, 2001). 

Methodology 

Participants 

28 volunteer (12 Females, 16 Males) 4th grade undergraduate students from the department of Computer 
Education and Instruction Technology at Middle East Technical University were the participants of this study. 14 
students (5 Female, 9 Male) were assigned to the group in which the participants experienced the signaling relevant 
keywords and 14 of them (7 Female, 7Male) were assigned to the group in which the participants experienced the 
signaling relevant sentences. Participants were between ages of 21 and 24 (Af=22.86, SD=. 71). Their Cumulative 
Grade Point Average (CGPA) were ranged from 1.86 to 3.84 (M= 2.86, SD= .51). They did not have prior knowledge 
about the topic of the material, because the topic of the material was about an imaginary country. 

Materials 
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The materials consisted of an informed consent letter, a retention test, and a transfer test, a computer-based 
material with signaled relevant keywords, a computer-based material with signaled relevant sentences and two 
computer-based practice materials. 

Informed consent letter was used to inform the participants about the study and to understand whether they 
participate voluntarily or not. It was one page paper-based material. 

Materials that were administered as paper-pencil tests consisted of the retention test and the transfer test, 
each on 21 cm X 29.7 cm sheets of paper. Participants could complete the materials in their own pace without any 
time restrictions. 

The retention test included five multiple-choice questions. This questionnaire assessed how much 
information the participants remembered about the population distribution of Patagonya according to the 
occupational sectors (“service”, “farming”, “industrial”) between 1970 and 2000. 

The transfer test consisted of eight open-ended questions. These questions assessed how the participants 
transfer their knowledge about the population distribution of Patagonya according to the occupational sectors 
between 1970 and 2000 to other environments. The questions were based on the comparison of the female and male 
distribution according to the sectors and comparison of the sectoral distribution. 

Computer-based practice materials included a graph which was about changing telephone tariffs between 
2004 and 2007 and an explanatory text below the graph. It was prepared by the researcher in Macromedia Flash 8. 
When the participant clicked one of the lines on the graph, the relevant keywords or sentences (based on 
experimental or control group) was underlined in the explanatory text. This material let the participants become 
familiar with the computer-based materials in the experiments. 

In the computer-based material in which the relevant sentences were signaled, there was a line graph about 
the population distribution of Patagonya according to the occupational sectors between 1970 and 2000 and an 
explanatory text below this graph. It was prepared by the researcher in Macromedia Flash 8. To maintain the 
content validity, sector names were taken from job statistics in the web site of Turkey statistical community. 
However, the line graph and explanatory text were designed by the researcher. When the participants clicked one of 
the line on the graph, the relevant sentences were signaled by using typographical cue, underlining. 

Computer-based material in which the relevant keywords were signaled was the same with computer-based 
material in which the relevant sentences were signaled. However, relevant keywords were underlined instead of 
relevant sentences when participants clicked one of the lines. 

Apparatus 

Eye movements of the participants were recorded by Tobii 1750 Eye Tracker, integrated in the panels of 
the monitor. The system had a 50 Hz sampling rate and an accuracy of 0.5°. 

Procedure 

The experiment was done in the Human-Computer Interaction laboratory. There was a Tobii eye tracker in 
this lab. It records the participants’ eye-movements during the experiment. Each participant was individually taken 
to this laboratory by an appointment. First of all, the informed consent letter was given to the participants and the 
researcher told them “Please read this letter and if you accept to participate in this study, please fill it and sign it”. 
After s/he signed it, the aim of the study was explained. Then, the participant used the computer-based practice 
material to become familiar with the computer-based material. Before conducting the actual experiment material, the 
automatic eye-tracking calibration of the subjects was done by Tobii’s ClearView software. The calibration gives 
information to the eye tracker of a particular subject’s eyes. It is required for “accurate estimation of a subject’s gaze 
point” (Tobii Technology AB, 2006). During the calibration, participants were presented a series of circles and 
requested to follow the circles by their eyes as the circles move within the screen. After the completion of the 
calibration process, a calibration plot was shown indicating the quality of the calibration by marking missing or 
inaccurate calibration points with red squares. If the collected calibration was poor, the calibration process was 
performed again. Next, participants studied the instructional materials. The participants were assigned to the group 
randomly. They studied the topic, the population distribution of Patagonya according to the occupational sectors 
between 1970 and 2000, by using the computer-based material in which the sentences were signaled or the 
keywords were signaled. There was no time limitation. They finished their study whenever they want. Study time 
changes from 90 seconds to 416 seconds. Participant’s eye movements were recorded by Tobii 1750 Eye Tracker 
while they studied the instructional material. The Clearview 2.7.1 program provided the duration of eye fixations, 
and the number of fixations. After finishing studying the instructional materials, each subject was given the retention 
test and the transfer test in respective order. 

Validity, Reliability, and Scoring 

In order to enhance the content validity, sector names were taken from job statistics in the web site of 
Turkey statistical community. The instruments were also pilot-tested with 10 students to evaluate the clarity of the 



13 




statements and to detect the usability of the computer-based materials. The researcher requested the participants of 
the piloting group to mark unclear statements and told about their interpretations. Revisions were made on these 
tests and computer-based instalments in accordance with the feedbacks from the pilot study. 

Retention test was assessed by givingl points for each correct answer. Transfer tests will be assessed by the 
researcher according to the 4-point scale rubric to minimize the measurement errors resulting from the rater’s 
judgment while scoring the open-ended questions (Penny, Johnson & Gordon, 2000). Another rater independently 
scored randomly selected transfer tests of 8 subjects. The raters were blind to the group of the answer sheet. Inter- 
rater reliability which indicates the agreements of the raters was calculated. The intra-class correlation coefficient 
was .82 indicating a high agreement of among the raters. Scores of the transfer test were converted to percentages. 
Data Analysis 

The results of retention and transfer tests and eye-movement data were used for data analysis. Independent 
t-test was conducted to understand whether there was a significant difference between the retention test scores of 
two groups of participants who study with the computer-based material in which the relevant sentences were 
signaled or in which the relevant keywords were signaled and whether there was a significant difference between the 
transfer test scores of two groups of participants who study with the computer-based material in which the relevant 
sentences are signaled or in which the relevant keywords are signaled. 

The material region was divided into different Area of Interests (AOI) according to the occupational 
sectors, keywords, sentences, graphs and texts. For data analysis, descriptive statistics and separate independent t- 
tests was used. 

Results 

Learning Outcomes 

Separate independent sample t-test was performed to examine whether signaling the relevant keywords or 
signaling the relevant sentences differed in retention and transfer tests. The participants who received the signaled 
relevant sentences material (M=58.57, £D=29.84) outperformed the participants who received the signaled relevant 
keywords material (M= 52.86, SD=2432) on the retention test. However, the effect of groups was not significant on 
retention, t (26) = .56, p=.58. Moreover, the difference in the transfer scores was not statistically significant between 
the learners who were administered the signaled relevant sentences material (M=13.50, SD=6.47) and the learners 
who were administered the signaled relevant keywords material (M=13.43, *SD=4.78), t (26) = .03, p=.91 . 
Eye-Tracking Data 

In order to calculate the average fixation duration on signaled sentence and keywords while sentence or 
keywords was underlined, and average fixation duration and average number of fixations until finding the relevant 
information, recordings of the participants were firstly divided into time segments by the intervals of the underlined 
sentences or keywords. An area of interest (AOI) was determined for the relevant sentence and keywords for each 
time interval. Thus, separate AOIs were created for each sentence and keyword. Finally, the sum of the eye-tracking 
measure was computed for all corresponding keywords and sentences. 

For the participants experienced the signaling relevant sentences, average fixation duration on the signaled 
sentence while the related sentence was underlined (M=l 18.79 ms, SD= 59.18) was very close to the average 
fixation duration on the keyword (M= 1 16.53 ms, SD= 66.08). On the other hand, while average fixation duration on 
the corresponding sentence was 108.10 ms (SD=55.90), it was 126.06 ms (£D=54.77) on the underlined keyword for 
the participants who experienced the signaled keywords. These results indicated that signaling increase the fixation 
duration on the signaled text. 

Average fixation duration until finding the relevant information in the text after clicking on the line was 
764.01 ms (SD= 309.69) for the group who experienced the signaling relevant sentences and 867.20 ms (SD= 

529.48) for the group who experienced the signaling relevant keywords. Moreover, for the participants who 
experienced in signaled sentences (M= 4.96 , SD= 1.54) made fewer fixation counts until finding the relevant 
information in the text after clicking on the line than those who experienced in signaled keywords (M= 4.76 , SD= 
2.33). However, the differences in average fixation duration and average fixation counts were not statistically 
significant according to the separate independent t-test results, t (26) = -,6l,p=.55 and t (26) = ,26,p=.80 
respectively. These results showed that signaling the keyword or signaling the sentence did not differ in taking 
learners’ attention to the relevant text according to both their finding time and searching. 

In the second part of the data analysis, the material region was divided into two AOIs which were graph 
and text. Whether there was a difference between fixation duration on text and graph regions was examined 
according to the frequencies. Average fixation duration on graphs (M=124.87 ms, SD=63.01) was higher than 
average fixation duration on text (M= 117.86, SD= 52.60). This shows that learners made a deep processing on the 
graph. Percentages of fixations made on the graph and on the text were examined. People made more fixations on 
the text part (Af=52.3%, SD=\ 7.7%) than on the figure part (M=47.7%, SD=\7 .7%). If unnecessary visual search is 
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responsible for cognitive load and consequently for impaired learning in split attention effect, then it is expected that 
the number of fixations would be smaller and average fixation duration would be longer on the graph part of the 
material. 

To understand how signaling effects on the overall reading, average fixation durations on keywords and 
sentences were compared according to the frequencies. To conduct this analysis, each keyword and sentence were 
selected as AOIs. The results showed that learners who experienced the signaling sentences had higher average 
fixation durations on sentences (M= 1 12.45, SD= 52.35) than on keywords (M= 105.96, SD= 69.06). On the other 
hand, learners who experienced the signaling keywords had higher average fixation durations on keywords (M= 
126.44, SD= 59.69) than on sentences (M= 116.28, SD= 48.02). These results showed that signaling directs the 
learners’ attention to the signaled text. However, according to independent sample t-test there was not a significant 
difference in average fixation duration on sentences and average fixation duration on keywords between the groups, 
t( 26)= .37, p=.85 and t( 26)= -.84,p=.41 respectively. This non-significant result might be explained by non- 
significant difference in their retention and transfer performance. 

Discussion & Conclusion 

The goal of this study was to investigate whether signaling the relevant keywords or signaling the relevant 
sentences was more effective on learning and to explore the effects of signaling on reading. The results suggested 
that there was not a significant difference between the retention and transfer scores of the learners who studied in the 
signaled sentences material and the learners who studied in the signaled keywords materials. This indicated that type 
of signaling does not make any change on the students’ performance. 

The effects of signaling on learners’ reading was further explored with the help of eye tracking data. 
Signaling had three main effects on reading. First, signaling increased readers’ attention to the specific information. 
People who were studying the material in which relevant sentences were signaled had higher average fixation 
durations on sentences whereas people who were studying the material in which relevant keyword were signaled had 
higher average fixation durations on keywords while the sentences or keywords were underlined. Moreover, for 
whole reading process, these results were the same. This showed that learners devote more processing time to 
underlined text. Second, signaling directed attention of the participants to the relevant information. Both group made 
little number of fixations and spent less time to find the information after clicking on the line. Third, signaling 
decreased unnecessary visual search, which was evident from longer average fixation duration and fewer number of 
fixations on graph. 

The effects of the signaling on reading supported the literature by giving clear evidence with eye- 
movements data. Lorch (1989) summarized the signaling effects as attention, selective case, discriminating 
important information and comprehension. Mautone and Mayer (2001) suggested that signaling is one of the 
solutions to reduce the extraneous cognitive load. 

Lorch (1989) stated signaling cause enhancing attention to the cued text. Thus, he suggested that reading 
time must be used to assess the attentional effect of signaling. Lorch et al. conducted a study to understand this 
effect. They found that reading speed was slower while the participants were reading a signaled text. This means 
that readers spent more processing time to signaled content than to nonsignaled content. 

According to the eye-tracking data, learners’ attention on sentences and keywords did not significantly 
differ. Furthermore, signaling the keyword or signaling the sentence did not differ in taking learners’ attention to 
the relevant text according to both their finding time and searching. Both signaled materials also decreased the 
visual search because average fixation count on graph was smaller and average fixation duration graph was longer. 
These non-significant results can be the reason of no significant difference in the students’ performance. 

Superior performance in signaled materials may be caused by tagging signaled content as important in 
memory without devoting more processing time to signaled information (Lorch, 1989; Lorch et al., 1995). Thus, in 
the future study, control group who studied the nonsignaled material can be used. By this way, their performance 
and their fixation duration will give compared results. Moreover, the sample of this study was 28 undergraduate 
students. Thus, this study should be replicated with a higher sample size. 

In addition to the limitation of sample size, the content of the material was not too long. For these reasons, 
no significant difference was found. Thus, the content should be longer in further studies. 

These results provide the following implications. To minimize the unnecessary search, designers should use 
the signaling devices in the materials in which both graphs and texts are used. By this way, learners match the 
related information for matching related information in the pictures that corresponds to the text. Instructional 
designers should be sensitive to the cognitive theory of multimedia learning while designing multimedia instruction. 
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Introduction 

One of the challenges of carrying out a qualitative research study is to develop coding categories when a 
priori categories do not exist. A number of sourcebooks describe different data analysis techniques (e.g., (Fraenkel 
& Wallen, 2006; Merriam, 1998; Miles & Huberman, 1994; Yin, 2003). This article focuses on the coding journey 
of one researcher, exploring how expert web visual designers carried out the preliminary stages of the visual design 
for a visual computer interface of an instructional website. 

Overview of Study Logistics 

This embedded case study (Yin, 2003) was exploratory (Stake, 1995) since it sought to discover a process 
not yet reflected in visual design research literature. The three participants worked individually. Each of them were 
presented with a simulation task (Fraenkel & Wallen, 1996) to stimulate the initial visual layout design process. 
Redesign Task 

The participants had the task of redesigning the visual look and feel for an existing website. In order to 
ensure authenticity, it was important for this task to reflect an appropriate, real-world design setting. An existing 
instructional website, entitled “The Great Computer Mystery,” was used as the basis of the simulation task. The 
website included a home page, intermediate navigation pages (one for each of four case-based scenarios or 
“mysteries”), and a series of content pages for each of the case-based scenarios (see Figure 1). 



Figure 1: Example of original "Great Computer Mystery" webpages 
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The redesign task targeted this website for a number of reasons. First, it was an authentic site, which had been used 
successfully: undergraduate students used the website in a computers-in-education course for two semesters, and the 
website won the Kemp Award for innovative instructional design. Second, the site had a very basic visual design. 
The webpages contained basic WordArt and clipart images created with Microsoft Office. Third, students used this 
site to work through case-based scenarios called “mysteries.” The researcher felt that the “mystery” metaphor also 
might pique the interest of participants in the study. 

Participants 

Three expert web visual designers were selected as participants in this study. Flaving multiple participants 
in the study brought richness to the data and allowed for cross-participant triangulation (Lincoln & Guba, 1985). In 
order to qualify as "experts" for this study (Durrance, 1998; Nonaka, 1991, , 1994; Polyani, 1967; Schon, 1983), 
participants had to meet the following criteria: (1) more than ten years of experience in graphic design, (2) more 
than four years of experience creating website look and feels, (3) design products that exemplify a professional level 
of visual design, and (4) the ability to describe their work in explicit terms. 

Data Collection 

The data collected included videotapes of the three sessions: the screening interview, the first design and stimulated- 
recall session, and the second design and stimulated-recall session, In addition to the videotaping, the data also 
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included sketches participants drew during the design sessions, transcripts of the videotapes, and computer files 
created during the sessions (see Figure 2). 




Figure 2: Types of data collected 
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Transcripts 

P: So then I thought, well what if I use a lamp, and 
actually I used an IU lamp, since we have all 
those great old lamps on campus, and had it go 
over far enough that you know, it kind of went 
that way and then there's the headline here (2 3. 
43) 

P: I thought okay, let’s start over Put all the type in 
one color And that’s when I switched it to the 
left side. 

I: So that’s what you’re doing right now with your 

switching it to the other side 

P: And for other reasons too. because we read left to 
right and. although I don’t know why I did it that 
way, maybe because I’m right handed. I don’t 
know (chuckle) So then I thought okay [drawing 
out the word okay with a tone of approval], 

(2.3. 53-58) 



volte it? P.l. Bolt iv ill help if 



Data Analysis 

The data coding process for this study involved the creation of coding categories, and the application of 
those coding categories to written transcripts and design actions. The six data coding phases were as follows: 

1. Creating Sentence-based Units 

2. Carrying out a Card Sort of Statements 

3. Coding of All Statements 

4. Writing Early Stories 

5. Coding Actions 

6. Mapping Participant Actions 



Creating Sentence-based Units 

To begin the coding process, the textual data was broken into sentence -based data units. Data units are 
defined by Merriam (1998) as follows: 

A unit of data is any meaningful (or potentially meaningful) segment of data. . ..A unit of data can be as 
small as a word a participant uses to describe a feeling or phenomenon, or as large as several pages of field 
notes describing a particular incident. 

According to Lincoln and Cuba (1985) a unit must meet two criteria. First, it should be heuristic — that is, 
the unit should reveal information relevant to the study and stimulate the reader to think beyond the particular bit of 
information. Second, the unit should be the smallest piece of information about something that can stand by itself — 
that is, it must be interpretable in the absence of any additional information other than a broad understanding of the 
context in which the inquiry is carried out. 

After reviewing the transcripts for the study, the researcher determined that the smallest heuristic unit size 
for the data was a sentence unit. Each sentence unit was identified by participant name, session number, and line 
number (see Figure 3) 
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Figure 3: Converting transcripts to numbered statements 
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JB. You know, when you land of adjust yourself. 
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I Gif just doesn't roll off bke. you're already saying jif or jpeg, 
for jig or jpeg, yeah 
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JB Did you sec the different programs they do have like a BB Edit 
type one called Home something or other homepage 




:* 


23 


JB And BB Edit is on here 





Carrying out a Card Sort of Statements 

To begin the data reduction or analysis of the data, it was necessary to develop context-sensitive coding 
categories (Merriam, 1998; Miles & Huberman, 1994; Schwandt, 1997). Following a card-sort procedure (Merriam, 
1998), the researcher printed each statement on a separate 2” x 2.5” card (see Figure 4). The researcher read each 
individual statement. If that statement had a similar meaning to a previously-read statement, they were clustered 
together. If not, it was put separately. Sometimes a statement would fit into more than one cluster. In this case, the 
researcher would write it on a separate card and put a copy of the card in each cluster. Sometimes a statement would 
shed a new insight to the clusters, and then clusters were adjusted to represent the new meaning structure. The 
researcher sorted the statements from one transcript using the card-sort procedure. At that point the creation of 
clusters had reached saturation, meaning there were not any significant changes to the clusters with additional 
statement coding. 

When all cards were sorted into clusters, the researcher selected a category name to represent the meaning 
of the statements within that cluster (see Figure 4). Sometimes the category name was a word used in the statements 
themselves. Sometimes the category name was a word that conveyed the meaning of the statements even though it 
was not directly mentioned. 
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The cluster names, resulting from the card sort, became the coding categories for data analysis. For ease of 
coding, the statements and coding categories were entered into a database (see Figure 5). Two coders, the researcher 
and an additional person, coded the statements from all the transcripts together. The researcher would read the 
statement aloud and determine the initial coding category for the statement. Meanwhile, the additional person 
would check off that code in a coding database and would double check the statement to see if she was in agreement 
with the researcher’s initial choice of code. If the additional person did not agree with the researcher’s choice of 
code, she woidd discuss the difference of opinion with the researcher, and they would both reach agreement as to the 
coding of the statement before making it final. The measure of inter-rater reliability for the initial coding categories 
of statements in the first transcript was 87%. However, there actually was 100% agreement for all final coding of 
statements due to the nature of the two-coder interaction. 
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During this data analysis stage, the coding categories were further refined (Miles & Huberman, 1994). 
When the data from a statement represented a new meaning, a new category was created to portray that meaning. In 
some cases a category would be split into separate categories that better reflected the nuances of the meanings. In 
other cases, two or more categories were combined because they had virtually the same meaning. At the end of this 
process, there were 113 coding categories. These categories were grouped into 13 category groups. This data 
reduction technique made it much easier to manipulate the data (Miles & Huberman, 1994) and to see the patterns 
within the 13 groups. 



Figure 5: Database entry screen with coding categories organized in category groups 
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Writing Early Stories 

Up to this point in the data analysis process, the researcher had focused on the transcript data containing 
statements. However, the researcher had not yet analyzed the actual sketches, files, and videotapes. Keeping the 
categories from the transcript analysis in mind, the researcher began coalescing the different types of data (see 
Figure 6) and describing the combined data through written stories or vignettes (Miles & Huberman, 1994). This 
process triangulated the statements with the other types of data (Glesne, 2005; Lincoln & Guba, 1985; Mathison, 
1999; Schwandt, 1997) and enabled the researcher to see how the pieces of data fit together. The result was a 
descriptive chronology that followed the sequence of what the participants did, as well as recording why they made 
specific decisions within the sequence. 



Figure 6: Writing early stories 



P(hyllis): Tlus is where I started 
I(ntemewer): So here you started oft' your sketch'’ 
P: Sketch, playing with the street light idea, which 
you it doesn't really pick up because it's pencil. 
And then also I had the notebook so I could see 
what typography had to go on the page, and I just 
wasn't sure if I could do half a hue. say m black, 
and the other half m wlute even (2.3, 31-33) 

Video Still Image 



Detail 



In order to test the feasibility of the combination of the graphic and text, 
she did a second sketch of the main menu page of the Web site (see Figure 3) 
that contained the same elements as the first sketch with one main exception— 
instead of using a placeholder for the submenu links, Phyllis wrote out the text. 
Phyllis was still not satisfied with this layout. As she put it, "I've dug myself 
into a hole here (2.3, 42). " She decided to abandon this layout for two reasons: 
(1) she felt that the audience would not be comfortable reading text with this 
type of optical illusion (2.3, 41) and (2) she considered that this treatment of 
mixed text colors might be limited by Web functionality (2.3, 40, 2.3, 52-53). 
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Coding Actions 

By writing the vignettes, the researcher became aware that the same coding categories used for the 
transcript statements could also be applied to coding the sketches, fdes, and videotaped actions. Next, the researcher 
created a database for the action coding process. The same coding categories were used; however, the category 
groupings were organized into types of actions and descriptors of the design (see Figure 7). The videotaped actions 
were coded by time index using this database. This enabled the researcher to find a common ground for coding 
what initially appeared to be vastly different types of actions. 



Figure 7: Grouped categories for detailed action coding 
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□ BUL bullets 
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Putting together detective, magnifying glass, and picture of school 



Next, the actions were compared using a summary worksheet in MS Excel (see Figure 8). During this 
coding phase, the researcher identified patterns of similar actions and calculated the duration of these patterned 
actions and the transitions from one patterned action to another. 
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Figure 8: Action summary worksheet 
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Mapping Participant Actions 



By enumerating durations of different types of actions, it was possible for the researcher to make a 
comparison map of the actions of the three different participants (see Figure 9). The comparison map reduced 
hundreds of actions into one visual representation (Miles & Huberman, 1994). This enabled the researcher to 
identify patterns and cross-participant, leading the petite-generalization and model-making processes. 



Figure 9: Comparison map of design actions across participants 
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Final Thoughts 

Overall, the coding process developed for this case study was very detailed. The early stage of carrying out 
the card sort of statements was time-consuming but necessary given the lack of a priori codes. The use of database 
and spreadsheet programs allowed the researcher to streamline the coding process, since the data were able to be 
manipulated in both formats. In addition, the coding categories developed through this analysis process may be used 
in future studies in the field of visual interface design. 
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Introduction 

An increased focus on problem solving and teamwork compels educators to engage learners in 
opportunities for knowledge seeking and collaboration. To achieve these goals, some educators are using 
constructivist pedagogical designs such as problem-based learning that are based on the assumption that learning is a 
product of both cognitive and social interactions in problem-solving environments (Evensen & Hmelo, 2000). 
Problem-based learning (PBL) is a promising instructional approach. However, it has its own share of challenges. 
When applied to real settings, PBL increases the complexity of the learning task and the cognitive load of students 
(Land, 2000). Scaffolding is proposed to address these challenges by providing support or imposing additional 
structure to learning in a PBL environment (Ge & Land, 2004; Pea, 2004). 

We are studying the impact of scaffolding along with other relevant individual characteristics in problem- 
based learning environments through a series of studies. The first study examined the effect of scaffolding and prior 
knowledge on individual posttest achievement, group project performance, student attitudes, cognitive load, 
navigation patterns and time in the program (Su, 2007). In this presentation, we will describe the second study in 
this series. 

Scaffolds 

In PBL environments, scaffolds are designed to induce learners to think and develop a deep and thorough 
understanding of the problem while arriving at a solution. One type, called metacognitive scaffolds, support the 
underlying processes associated with individual learning, providing guidance on how to think during learning. 
Metacognition can be best explained as thinking about thinking. Land (2000) outlines three metacognitive skills 
required in open ended learning environments such as PBL - (a) identifying appropriate learning issues, (b) 
monitoring the effectiveness of learning strategies and (c) monitoring detailed steps in problem-solving while 
remaining focused on the task. Land (2000) further claims that many students lack these metacognitive skills and 
that supports should be designed to help them. Metacognitive scaffolds introduced in this study were designed to 
facilitate learners to plan, monitor, regulate and reflect upon their learning process. 

Self-directedness 

The role of self-directedness in an individual’s learning process has been extensively studied. Though the 
definition of self-directed learning varies throughout the literature, the most common definition is that of Knowles 
(1975), who defined self-directed learning as a process in which individuals take the initiative, with or without the 
help of others, to diagnose their learning needs, formulate learning goals, identify human and material resources for 
learning, choose and implement appropriate learning strategies, and evaluate learning outcomes. Self-directedness 
has been shown to be a strong factor in predicting academic achievement (Hsu & Shuie, 2005). Self-directed 
learning is also an integral part of the PBL instructional approach (Hmelo-Silver, 2004). A new tool was developed 
to study self-directedness in this study. Prior to the research study, participants were each administered a “Learning 
Notions and Preferences Survey (LNPS)” which collected data describing an individual’s inclination to be engaged 
in a self-directed study. Responses to this survey were assessed on a five-point Likert-type scale and converted to a 
score. This score was used to evaluate an individual’s self-directedness. 

The purpose of this study was to investigate the impact of metacognitive scaffolds and self-directedness on 
project performance, participant achievement and attitudes within the context of a computer-mediated, collaborative 
problem-based learning environment. The main research questions addressed in this study were: 

1 . What is the effect of metacognitive scaffolds on participant achievement and attitude, in a 

computer-mediated, collaborative problem-based learning environment? 

2. What is the relationship between self-directedness and participant achievement and attitudes in a 

computer-mediated, collaborative problem-based learning environment? 
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Method 



Design & Participants 

This study used a pretest-posttest experimental research design. The independent variables were 
metacognitive scaffolds (absence or presence) and self-directedness. The dependent variables were group project 
performance, individual posttest achievement, and participant attitudes. 

Participants were 39 male and female undergraduate college students enrolled in an educational technology 
course at a large southwestern university in the United States. The course was offered through the College of 
Teacher Education and Leadership. Participants received extra course credit for their participation. They were 
grouped into teams of three and the teams were randomly assigned to one of the two treatment groups (No 
Metacognitive Scaffolds - NMS or Metacognitive Scaffolds - MS). 

Materials 

The instructional material used in this study was a web-delivered, hypermedia problem-based instructional 
program called “All you need is a screwdriver” developed by Su (2007). Central to the program was a PBL project 
of building a virtual desktop computer achieved through collaborative effort. As is typical of PBL environments, this 
project was designed as an ill-defined problem. Built into the program were resources for information and guidelines 
for component selection. Content scaffolds, which included a warm-up sheet, a note taking sheet and a project 
template were common to both treatment groups and were designed to support participants’ understanding of the 
content. 

Participants in the NMS group received instructional materials with only the content scaffolds. Three 
metacognitive scaffolds (MS) were available to participants in the MS treatment group in this study: (1) a project 
planning sheet for each team, (2) a project reflection sheet for each team member, and (3) software cues in the form 
of pop-ups that urge the participants to reflect on their choices before making a final decision of component 
selection. 

The objective of the project was for project teams to build a virtual computer that could be used for tasks 
such as doing homework, listening to music, viewing videos, surfing the internet, downloading files and playing 3-D 
video games within a prescribed budget of $1000. The project did not have a unique answer. A team’s answer could 
be the solution to the problem as long as the proposed virtual computer configuration met the performance, 
compatibility and budget requirements. 

Data Sources 

Five data sources were used in the study. These include a pretest, a posttest, the LNPS score, an attitude 
survey and a group project score. The test items and scoring rubrics for the pretest, posttest, group project and the 
attitude survey were borrowed with permission from Su (2007) and modified to meet the needs of the current study 
by the primary researcher. 

Pretest. Prior to completing the instructional materials, a 12-item, computer-based, four-choice, selected 
response pretest measuring general computer knowledge was administered to the participants. This pretest provided 
data about the general proficiency and comfort level of participants in the areas of computer components, hardware 
and connectivity. Each pretest item was worth one point. All pretests were automatically scored by the computer 
program and the results downloaded to a database. 

Posttest. Following the completion of the PBL project, a 20-item, computer-based, four-choice, selected 
response posttest measuring individual knowledge and skills covered in the instructional program was administered. 
The test items were generally aligned to the project objectives to establish content validity. Each posttest item was 
worth one point. All posttests were automatically scored by the computer program and the results were downloaded 
to a database. 

LNPS score. The Learning Notions and Preferences Survey (LNPS) score was based on the responses to a 
self-reported, 18-item, computer-based instrument developed by the primary researcher. This instrument measured 
an individual’s inclination to be self-directed Participants responded to questions designed to elicit individual 
feelings about their capacity of performing a task specifically focusing on their previous experience. The items were 
rated on a five-point Likert scale with “SA” representing “Strongly agree”; “A” =Agree; “U” = Undecided; “D” = 
Disagree, and “SD” = Strongly disagree. The response “SA” corresponds to “5 points”; “A” = 4 points; “U” = 3 
points; “D” = 2 points, and “SD” = 1 point. The maximum possible LNPS score was 90 and the minimum was 18. 
This survey was administered at the beginning of the research study before the pretest. 

Attitude survey. An attitude survey was developed to measure participants’ reactions to all aspects of the 
instructional program. The survey was administered immediately after the posttest on the day of the study. It 
contained 16, five-choice Likert-type (SA-strongly agree, SA-strongly disagree) and two open-ended questions. The 
survey included questions to elicit participant responses about the characteristics of the instructional program, 
working in small groups, problem-solving approaches and perception towards scaffolds. 
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Group project score. The project score was assessed using a scoring rubric. The rubric primarily assessed 
the accuracy of the answers and the quality of the rationale descriptions. The rubric was designed to consider all 
aspects of the problem-solving process and provided for the possibility of awarding partial credit for partially correct 
answers. The total possible score for the project was 55 points. The primary researcher scored all the group projects. 

Procedures 

The study was conducted in regularly scheduled classrooms with individual computers. On the day of the 
study, the primary researcher introduced the instructional unit to the participants. Participants were informed about 
the nature of the study before agreeing to participate and were assured that their participation in the study was 
entirely voluntary. They were also informed that all individual results were confidential and not available to anyone 
except the research team, and that any data reported would be aggregated. Participants were notified that they would 
be working in groups on an online program to build a virtual computer in two hours and that they were not allowed 
to access any other external site except the online instructional program website during the research study. The 
regular classroom instructor then told the participants that their full participation in the study would earn them five 
extra credit points in the course. Initially, the individual, computer-based LNPS survey was administered to all 
participants. This was immediately followed by the pretest. Subsequently, teams of three were formed based on 
existing seating in the classroom and the teams were randomly assigned to one of the two treatment groups. Each 
team was then handed an envelope that contained the study materials corresponding to the assigned treatment group. 
Participants were then directed to the instructional program website and instructed to begin their work. During this 
phase of the study, the teams followed the instructions provided on the instructional program website to complete 
the project. At the conclusion of two hours all program materials were assembled, inserted into the envelope and 
returned to the primary researcher. The individual computer-based multiple-choice posttest was then administered to 
each participant. Upon completion of the posttest, individual participants also completed a Likert-type attitude 
survey. 

Data Analysis 

SPSS version 17.0 was used for all data analysis. Two major research questions were addressed in this 
study: 1) What is the effect of metacognitive scaffolds on participant achievement and attitude, in a computer- 
mediated, collaborative problem-based learning environment, and 2) what is the relationship between self- 
directedness and participant achievement and attitudes in a computer-mediated, collaborative problem-based 
learning environment. Table 1 conveys the descriptive statistics associated with each of the independent and 
dependent variables. The table reports the mean scores and standard deviations of each condition (No Metacognitive 
Scaffolds - NMS or Metacognitive Scaffolds - MS) on the pretest, LNPS score, posttest and project score. Two 
separate one-way analyses of covariance (ANCOVAs) were conducted to examine the effect of metacognitive 
scaffolds and LNPS score on the posttest and project performance, using the pretest score as a covariate, each at 
alpha = 0.05. Preliminary analyses were conducted to test for the homogeneity of slopes between the covariate 
(pretest score) and the dependent variable (posttest score or project performance score) across groups (No 
Metacognitive Scaffolds - NMS or Metacognitive Scaffolds - MS). All means reported in this section were adjusted 
for the covariate. 

Results 

Project performance. Unadjusted means and standard deviations for this task are reported in Table 1. The 
average score for all project teams was 37.99 (SD=5.36) out of a possible score of 55. The mean score for teams in 
the No Metacognitive scaffolds group (NMS) was 34.37 (SD=3.71) and for teams in the Metacognitive scaffolds 
Group was 41.61 (SD=4.20) 
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Table 1: Unadjusted mean scores and standard deviations by treatment type on project score. 



Treatment type 


Project Score 


No Metacognitive Scaffolds (NMS) 




Mean 


34.37 


SD 


3.71 


N 


23 


Metacognitive Scaffolds (MS) 




Mean 


41.61 


SD 


4.20 


N 


23 


Total 




Mean 


37.99 


SD 


5.36 


N 


46 



A preliminary analysis of the homogeneity-of-slopes assumption indicated that the relationship between the 
covariate (pretest) and the dependable variable (project performance score) did not vary significantly as a function 
of the independent variable (treatment type), F(l,42) = 1.03, MSE = 16.02 ,p = .32, partial // 2 = -16. A one way 
analysis of covariance (ANCOVA) was conducted to examine the effect of metacognitive scaffolds on project 
performance using pretest as a covariate. The ANCOVA was significant, ^(1 ,43) = 35.45, MSE = 16.03 ,p <.01. The 
strength of the relationship between the metacognitive scaffolds group and the project performance score was very 
strong as assessed by a partial T ] 2 , with the metacognitive scaffolds factor accounting for 45% of the variance of the 
project performance score, holding the pretest score constant. The Metacognitive Scaffolds group (MS) had the 
largest adjusted mean (M= 41.58) and the No Metacognitive Scaffolds group had the smallest (M= 34.40). 
Furthermore, a one way analysis of covariance (ANCOVA) conducted to examine the relationship between self- 
directedness (as measured by the LNPS score) on project performance using pretest as a covariate did not yield a 
significance difference. 

Posttest. Unadjusted means and standard deviations for this task are reported in Table 2. The average score 
for all individual posttests was 1 1.37 (SD=3.02) out of a possible score of 20. The mean score for individuals in the 
No Metacognitive scaffolds group (NMS) was 1 1.04(SD=3.20) and for individuals in the Metacognitive scaffolds 
Group was 11. 70(SD=2.87). 
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Table 2: Unadjusted mean scores and standard deviations by treatment type on posttest score. 



Treatment type 


Posttest score 


No Metacognitive Scaffolds(NMS) 




Mean 


11.04 


SD 


3.20 


N 


23 


Metacognitive Scaffolds(MS) 




Mean 


11.70 


SD 


2.87 


N 


23 


Total 




Mean 


11.37 


SD 


3.02 


N 


46 



Posttest performance was analyzed by treatment group and self-directedness score using pretest score as a 
covariate to determine differences in achievement. The ANCOVA analysis yielded no significant differences either 
for the treatment group or the self-directedness score on project performance. 

Attitudes. Responses to 16 Likert-type items on the computer-based attitude survey were scored on a 5 
(strongly agree) to 1 (strongly disagree) basis. The mean attitude scores and standard deviations by treatment group 
are shown in Table 3. These data indicate that most participants felt that their group distributed their work load 
fairly among its team members (M = 4.21, SD = .92) and that their group designed the computer using the 
information obtained from their research in the program (M = 3.82, SD = .76). The data also suggests that the 
participant groups had enough time to complete the project (M = 3.77, SD = 1.16). However, participant responses 
were not as positive to the item that asked whether they liked the computer program (M = 2.59, SD = 1.14). 
Participants also indicated that they would probably not enjoy working on another similar program (M = 2.69, SD = 
1.22) and that they were not confident that they would pass the individual test (M = 2.69, SD = 1.10). 

The attitude data were analyzed using MANOVA to determine whether there was a significant difference 
between the mean scores of treatment groups and self-directedness score (LNPS score) on the survey items. No 
significant effects were found either for the treatment group or for self-directedness. 

Approximately 78% of the study participants who completed the Likert-type portion of the attitude survey 
responded to either one or both of the open-ended questions. When asked what they liked most about the program 
18 out of 29 respondents (62%) mentioned that they got to learn more about computers. For example, one 
respondent stated that “they got to learn how a computer is built and what goes into it.” Further, 12 out of 29 
respondents (41%) indicated that they enjoyed the group aspect of the program. One responded notes ‘I liked that 
you allowed us to work in groups rather than as individuals. This allowed for us to communicate about different 
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ideas together.” When asked how the program could be improved 11 out of 29 respondents (38%) suggested that the 
length of the program be reduced, 6 respondents (21%) indicated that the program was hard and 5 (17%) 
respondents expressed that the project required a lot of writing. 



Table 3: Attitude scores by treatment group. 







Scaffolding 






Items 


NMS 


MS 


Total 


1. 


I liked the “All you need is a Screwdriver” 


2.72 


2.48 


2.59 




program. 


(1.11) 


(1.25) 


(1.14) 


2. 


1 learned a lot about computer hardware 


3.39 


3.43 


3.41 




from this program. 


(.85) 


(1.08) 


(.97) 


3. 


The program was well designed. 


3.44 


3.43 


3.44 






(.71) 


(1.08) 


(.91) 


4. 


I would enjoy working on another project 


2.83 


2.57 


2.69 




like this again. 


(.92) 


(1.43) 


(1.22) 


5. 


I am confident that I passed the individual 


2.78 


2.62 


2.69 




test at the end of the lesson. 


(.73) 


(1.36) 


(1.10) 


6. 


My group had enough time to complete the 


3.72 


3.81 


3.77 




project. 


(1.23) 


(1.12) 


(1.16) 


7. 


My group planned how to approach the 


3.61 


3.38 


3.49 




project before we got started on it. 


(1.04) 


(1.12) 


(1.07) 


8. 


My group distributed our work load fairly 


4.39 


4.05 


4.21 




among team members. 


(.78) 


(1.02) 


(.92) 


9. 


My group designed the computer using 


3.89 


3.76 


3.82 




information obtained from our research in 
the program. 


(.68) 


(.83) 


(.76) 
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10. Working with my team helped me do well 


3.50 


3.33 


3.41 


on the individual test at the end of the 
lesson. 


(.99) 


(.86) 


(.91) 


1 1 . The program included enough information 


3.61 


3.24 


3.41 


and resources to help me do well on the 
individual test at the end of the lesson. 


(1.04) 


(.10) 


(.97) 


12. The program offered enough support to help 


3.72 


3.43 


3.56 


my team organize our project. 


(.83) 


(1.12) 


(.10) 


13. It was easy to find information needed to 


3.61 


3.38 


3.49 


complete the project. 


(.92) 


(1.12) 


(1.12) 


14. The program provided enough support at the 


3.83 


3.52 


3.67 


beginning to help my team get started on the 
project. 


(.86) 


(1.03) 


(-96) 


15. The program included enough support to 


3.89 


3.48 


3.67 


help me keep track of information I 
collected to complete the project. 


(.47) 


(1.08) 


(.87) 


16. My group reflected on what we learned from 


3.17 


3.14 


3.15 


the program. 


(.79) 


(l.H) 


(.96) 



Note. Scores are based on a 5-point scale (5 - strongly agree, 1 - strongly disagree). 



Discussion 

This study examined the effect of metacognitive scaffolds on achievement and attitudes of university 
undergraduate students. It also explored the relationship between self-directedness (as indicated by the LNPS score) 
and participant achievement and attitudes. Participants in the metacognitive scaffolds (MS) group scored 
significantly higher on the group project score than those in the no metacognitive scaffolds (NMS) group. 

Group project 

The significantly higher group project score for participants in the MS group over those in the NMS group 
is most likely because of the presence of metacognitive scaffolds. This study incorporated three metacognitive 
scaffolds: (1) a project planning sheet for each team - to help participants identify how a learning task can be 
divided into sub-tasks and approached (Quintana, Zhang, & Krajcik, 2005) (2) a project reflection sheet for each 
member - to encourage participants to summarize the collected information, reflect on, and debrief information they 
learned after finishing the project and (3) software cues in the form of pop-ups that urge the participants to reflect on 
their choices before making a final decision of component selection (Su, 2007). These findings are in line with those 
by Wolf (2000) who reported that students in the metacognitive scaffolds group outperformed those with no 
metacognitive scaffolds. However, unlike this study, students in Wolf s study did not receive any content scaffolds. 
The finding that self-directedness (LNPS score) did not yield a significant difference over the no metacognitive 
scaffolds (NMS) group for group project performance may be due to the high scores of all participants on self- 
reported LNPS survey ( M = 71.6, SD = 6.55) over a total possible score of 90 points. 

Posttest 

The finding that the metacognitive scaffolds (MS) group did not yield a significant difference over the no 
metacognitive scaffolds (NMS) group for posttest achievement is inconsistent with the findings of Su (2000). A 
factor that could have mitigated the potential overall effect of metacognitive scaffolds in this study is the presence of 
content scaffolds for both the treatment groups. Content scaffolds in this study directed participants to important 
concepts and rules by asking specific and detailed questions. For example: correctly answering the content scaffold 
question “What is RPM? How does it affect hard drive performance?” enabled the participant to answer the 
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following posttest item: “Which of the following items correctly describes a hard drive performance?” Answering 
questions posed by content scaffolds enabled students to acquire specific knowledge which was required on the 
individual posttest (Su, 2000). The finding that self-directedness (LNPS score) did not yield a significant difference 
over the no metacognitive scaffolds (NMS) group for posttest performance may be due to the possibility that the 
LNPS score may not be a powerful enough factor to yield significant difference in posttest achievement scores. 

Attitudes 

Overall, participants expressed near neutral attitudes towards most features of the program. Respondent 
scores averaged between 2.6 to 3.7 on a 5-point Likert-type scale (3 = neither agree nor disagree). Survey questions 
that dealt with group work elicited most favorable responses while questions about the program topic and content 
were returned neutral responses. An analysis of the participants’ response to the open-ended items of the survey may 
explain low scores on some of the items. While responding to the open-ended questions, many participants felt that 
the program was too long. Some felt that the content itself was hard to comprehend and that there was a lot of 
writing involved. These reasons may have impacted participants’ attitude towards the PBL program. The attitude 
survey did reveal some positive things about the program. Many enjoyed the new information presented in the 
program, working in groups and the design of the program. These responses indicate that the PBL module was 
perceived as both useful and valuable. 

Implications and Further Research 

As a second part of a series of studies of the impact of scaffolding along with other relevant individual 
characteristics in problem-based learning environments, we have presented our findings from the study of the impact 
of metacognitive scaffolds and self-directedness in Computer-mediated Problem-based Learning. We feel that the 
present study has implications both for educational practice and further research. With regards to practice, the study 
indicates that the use of metacognitive scaffolds together with content scaffolds is likely to improve group project 
performance scores. With regards to research, the present study does not provide clear answers to the issue of 
whether self-directedness might impact achievement when stuents use PBL. Future research should continue to 
study PBL and scaffolds. Additional studies are required to isolate significant effects of different types of scaffolds 
and other individual characteristics. As complex PBL environments become mainstream, researchers should 
continue to investigate factors that ensure participant success. 

References 

Evensen, D. H., & Hmelo-Silver, C. E. (2000). Problem-based learning : A research perspective on 
learning interactions. Mahwah, N.J.: L. Erlbaum Associates. 

Ge, X., & Land, S. M. (2004). A conceptual framework for scaffolding ill-structured problem-solving 
processes using question prompts and peer interactions. Educational Technology’ Research and Development, 52(2), 
5-22. 

Hmelo-Silver, C. (2004). Problem-based learning: What and how do students learn? Educational 
Psychology’ Review, 16(3), 235-266. 

Hsu, Y., & Shiue, Y. (2005). The effect of self-directed learning readiness on achievement comparing face- 
to-face and two-way distance learning instruction. International Journal of Instructional Media, 32(2), 143. 

Knowles, M. S. (1975). Self-directed learning :A guide for learners and teachers. Chicago: Association 

Press. 

Land, S. M. (2000). Cognitive requirements for learning with open-ended learning environments. 
Educational Technology’ Research & Development, 48(3), 61-78. 

Pea, R.D. (2004). The social and technological dimensions of scaffolding and related theoretical concepts 
for learning, education, and human activity. Journal of the Learning Sciences, 13, 423-451. 

Quintana, C., Zhang, M., & Krajcik, J. (2005). A framework for supporting metacognitive aspects of online 
inquiry through software-based scaffolding. Educational Psychologist, 40(4), 235-244. 

Su, Y. The impact of scaffolding type and prior knowledge in a hypermedia, problem-based learning 
environment. Ph.D. dissertation, Arizona State University, United States — Arizona. Retrieved October 3, 2009, 
from Dissertations & Theses @ Arizona State University. (Publication No. AAT 3288016). 

Williams, B. (2004). Self direction in a problem based learning program. Nurse Education Today, 24(4), 

277-285. 

Wolf, Sara Elizabeth (2000). The Big Six Information Skills as a metacognitive scaffold in solving 
information-based problems. Ph.D. dissertation, Arizona State University, United States — Arizona. Retrieved 
October 3, 2009, from Dissertations & Theses @ Arizona State University. (Publication No. AAT 9976355). 



33 




Exhuming Cultural Artifacts to Embed and Integrate Deep Adult E-Learning 

Ruth Gannon Cook, Ed.D. 

DePaul University 

Caroline Crawford, Ed.D. 

University of Houston Clear Lake 

Abstract 

This study addresses which factors can have a positive influence on adult on-line learners by studying 
cognitive load, the use of technology and semiotic tools in on-line courses. It incorporates theory and research from 
multiple disciplines and weaves cultural artifacts into e-learning based on the findings of a survey that looked at 
which factors worked successfully to transform e-learning into integrated learning for adult learners. The findings 
may provide important insights into student cognitive load and e-learning that can be extended to other colleges 
contemplating offering on-line courses and programs. 

Purpose of the Study 

The purpose for the study was to gather data on students at a private and a public urban university that both 
had students enrolled in on-line courses and hybrid degree programs in order to assess the cognitive load and quality 
of learning. The reason for the assessment of cognitive load is because it is the study of the mental processes of 
learning, memory and problem solving. 

Cognitive load theory (e.g. Sweller, 1988; 1994) is an instructional 
theory. . .(that) describes learning structures in terms of an information 
processing system involving long term memory. . . Information 
may only be stored in long term memory after first being attended to, 
and processed by working memory... The fundamental tenet of 
cognitive load theory is that the quality of instructional design will 
be raised if greater consideration is given to the role and limitations, 
of working memory. (Cooper, 1998, p.l) 

The research was conducted to see if the retention rates of students in these courses were higher with the 
addition of sociocultural tools and artifacts than the courses without these additions. The researchers were also 
interested to see if the university rate of attrition was less, equal to, or more than the national average of student 
attritions in on-line programs at other universities in the United States. The intent was to observe which factors 
could positively influence cognitive load and student retention, particularly in view of the fact that the national 
average attrition rate of students in on-line courses in 2002 was approximately 40% (Hudson & Shafer, 2005; 
National Center for Educational Statistics Report, 2002). 

The use of interactive multimedia in on-line courses allows the integration of multiple sources of 
information and tool mediation (Ambron & Hooper, 1990; Cooper, 1998; Crawford, GannonCook, & Rudnicki, 
2003; Driscoll, 2000; Nelson, Ketelhut, Clarke, Bowman, & Dede, 2005; Nersessian, 2007; Vygotsky, 1962, 1978, 
1981;Wertsch, 1985). However, effective student use of more complex multimedia into course materials (more 
complex by the fact that the student not only must leam the course materials, but also how to use the course 
navigation and technology) depends on the way the student processes data and how much cognitive load the student 
can handle before she or he becomes saturated or overwhelmed with data (Huffman, Goldberg, & Michlin, 2003; 
Kegan, 1993). If the student, particularly an adult learner (24 years or older), is coming into college after being out 
of school for a while, a task with two or three simple steps on- and off-line could cause frustration and negative 
reaction. And it is here that the student must scaffold her or his new course content materials. 

The need to study cognitive load in on-line learning, particularly because of the limitations of working 
memory, must be considered when certain factors, such as the need to know how to use technology and navigate 
course materials, will remain a challenge for many students new to or uncomfortable (because of learning styles, 
self-regulation skills, or cultural differences) with on-line learning. However, while there are maximum limits of 
working memory, there is also evidence that expansion of working memory may be possible (Pavio, 1990; 

Baddeley, 1992; Basak, n.d.). So, in addition to studying factors that could reduce cognitive load, ways to expand 
working memory can also be pursued as a means of facilitating on-line learning (Cooper, 1998). By studying what 
facilitates and frustrates on-line learning, strategies for effective knowledge transfer could reduce cognitive over- 
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load and help learners to acquire and assimilate course materials in ways that reinforce learning and course 
completion (Cooper, 1998). 

Background 

Adult learners, for the purposes of this study, are defined as students 24 years or older (National Center for 
Education Statistics, 2002). and mirrors the national average age and demographics of students in on-line courses 
(most work full-time while attending college) (Berker & Horn, 2004). While a majority of undergraduates in 
traditional degree programs were younger than 24, one in four students was actually 30 or older, or about 43 percent 
of undergraduates who were enrolled in 1999-2000 were age 24 or older (Bradburn, Berger, Li, Peter, & Rooney, 
2004). 

Research on adult-learners in on-line courses indicates that the majority of post-secondary on-line courses 
have high attrition rates (National Center for Education Statistics, 2002), as high as forty percent or more. While 
there may be a number of reasons for this high attrition, one of the reasons may be related to cognitive over-load. 
This study looked at cognitive load, including the learning curves of technology, time management, and new content 
materials, to see what effect they might have on both student attitudes and retention in on-line courses. It also looked 
at working memory (WM) limitations (Mayer, 2001, 2005), how those could be minimized, how WM could be 
expanded, and how there could be “learning consequences for ignoring those limitations” (p.22). 

Prior research indicated that while there are maximum limits of working memory, there was also evidence 
that expansion of working memory might be possible (Bailey, 1999; Basak, n.d.; Birdsong, 1999; Jewitt & Oyama, 
2001; Mayer, 2001; Sweller, 1988; 1994). 

Methodology 

The study was designed to interweave theory, research, practice, and theory development and explore 
whether there could be inferences as to which factors could influence cognitive load and student retention in online 
courses. The researchers’ made efforts to actively incorporate methodologies from multiple methodologies, such as 
experimental research using a Likert-type of survey and developmental research (McKenney & van den Akker, 

2005; Richey, Klein, 2007) appreciative inquiry (Preskill, Catsambas, 2006), and action research (Singleton & 
Straits, 2005). The survey sampled 109 adult learners to analyze data from an electronic survey that was designed to 
include elements of appreciative inquiry. One of the researchers was also embedded in the online course to 
document events and activities that occurred during the study. 

The study researched adult-learner cognitive load in on-line courses and compared text-based online 
courses (traditional courses transferred to on-line formats without graphics, metaphors, or multimedia) and graphics 
and media-enhanced on-line courses (courses designed with embedded semiotics, such graphics and metaphors, or 
multimedia) to see which had a more positive effect on adult-learner cognitive load and subsequent learner retention. 
Prior research indicated that the inclusion of multi-media into online courses could enrich student learning (Bailey, 
1999; Beggs, 2002; Brooks, 2008; Cobley, Jancz, 1997; Crawford, Gannon-Cook, 2008; Gannon-Cook, 2006; 
Gannon-Cook, Crawford, 2006, 2007; Greeno & Hall, 1997; Mayer, 2001; Ryan, Hodson-Carlton, & Ali, 2005), the 
findings of the study supported those findings (Hard & Jamison, 2005). In addition, the study found that the factors 
of embedded graphics and metaphors, followed by the integration of visual multimedia, such as embedded multi- 
media PowerPoints, or MP4s, were more effective than some other multi-media, such as Wimba or Second Life to 
lessen cognitive load and best predict student retention for adult learners. The students expressed recognition of 
simple signs and icons, and acknowledged that these did serve as kind of signposts that continued to point them in 
an onward direction through the course. 

One of the biggest surprises in the researchers’ findings was how much the students seemed to get engaged 
in telling their own stories. (Narratives were embedded in the modules to function as yet another form of semiotics). 
The vast majority of students expressed the most satisfaction from reading about and sharing stories and experiences 
with each other. Both researchers were genuinely surprised that the cultural icons and tools seemed to help the 
students to stay focused; they seemed to open up a bit more in the modules with the icons and indicate a willingness 
to communicate more about their lives. They also seemed to stay involved and expanded their online discussions 
beyond their assignments in many instances with only two students dropping out of the course and those two took 
incompletes. 

Summary 

New technologies are often implemented by colleges as a kind of panacea to stay competitive with other 
universities or as remedies for problems such as attrition. These reasons can be valid, particularly as a competitive 
tool, but there may be other factors that could provide deep solutions that could both facilitate integration of new 
technologies seamlessly into the fabric of their learning, and help students weave new content knowledge into their 
daily lives. 
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This study was conducted to see how student cognitive load could be minimized and monitored so that 
new technologies could be integrated seamlessly into learning, instruction and performance. The study provided 
insights into which factors could help regulate cognitive load for students. Further studies could provide more data 
on how these factors could reduce student attrition in online courses. 
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This study addresses which factors can 
have a positive influence on adult on-line 
learners by studying 




• the use of technology, and 

• semiotic tools in on-line courses 
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The Purpose of the Study 

• to gather data on students at a private and a 
public urban university that both had 
students enrolled in on-line and hybrid 
degree programs to see what may positively 
influence the students, help them feel 
encouraged and nudge them to finish the 
course. 



Cognitive load theory 

(e.g. Sweller, 1988; 1994) 

• Information may only be stored in 

long term memory after first being attended to, 
and processed by working memory... 

• The fundamental tenet of 
cognitive load theory is that 

the quality of instructional design will be raised 
if greater consideration is given to the role and 
limitations, of working memory. (Cooper, 1998, p.i) 
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• The research was conducted to see if the 
retention rates of students in these programs 
were higher with the addition of semiotic 
sociocultural tools and artifacts than the 
courses without these additions. 
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Effe^iye student use of more complex ideas 
diwmullimedia imonline course materials 



depends on the way the student processes 

data and j 

how much cognitive load the student can 
hpidle berore she or he becomes Saturated 
or overwhelmed with data (Huffman Goldberg, & 

■liEhlin, 2003; Kegan, 1993). 




• So, in addition to studying factors that could 
reduce cognitive load, 

• ways to expand working memory can also be 
pursued as a means of facilitating on-line 
learning (cooper, 1998). 

• By studying what facilitates and frustrates on- 
line learning, strategies for effective knowledge 
transfer could reduce cognitive over-load and 

• help learners to acquire and assimilate course 
materials in ways that reinforce learning and 
course completion (cooper, 1998). 
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• Research on adult-learners in on-line courses 
indicates that the majority of post-secondary 
on-line courses have high attrition rates as 
high as forty percent or more 

(National Center for Education Statistics, 2002). 



While there may be a number of reasons for 
this high attrition, one of the reasons could 
be related to cognitive over-load. 



It 





This study looked ,at cognitive load, 

including the ^ learning curves 

of technology, Hi time management, 

and trying to learn new content materials, to see what 
effect they might have on both student attitudes and 
retention in on-line courses. 



• It also looked at working memory (WM) limitations 

(Mayer, 2001, 2005), and 

• how those could be minimized, 

• how WM could be expanded, and 

• what "learning consequences for ignoring those 
limitations" might occur (p.22). 
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• Prior research indicated that while there are 
maximum limits of working memory, there 
was also evidence that expansion of working 
memory might be possible (Bailey, 1999; 
Basak, n.d.; Birdsong, 1999; Jewitt & Oyama, 
2001; Mayer, 2001; Sweller, 1988; 1994). 



• The researchers' made an effort to actively 
incorporate multiple methodologies, such as 

• experimental research using a Likert-type of 
survey, 

• developmental research (McKenney & van den 

Akker,2005; Richey, Klein, 2007) 

• appreciative inquiry (Preskill, Catsam; Singleton & Straits, 
2005; Basak, n.d.), and 

• action research (Richey, and Klein, 2007). 




Methodology 
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• The study researched adult-learner cognitive load 
in on-line courses and compared text-based online 
courses (traditional courses transferred to on-line 
formats without graphics, metaphors, or 
multimedia) and 



graphics and media-enhanced 
on-line courses (courses designed 
with embedded semiotics, such 
graphics and metaphors, or 
multimedia) 

to see which had a more positive 
effect on adult-learner cognition. 




• Prior research indicated that the inclusion of 
multi-media into online courses could enrich 

Student learning (Bailey, 1999; Beggs, 2002; Brooks, 

2008; Cobley, Jancz, 1997; Crawford, Gannon-Cook, 2008; 
Gannon-Cook, 2006; Gannon-Cook, Crawford, 2006, 2007; Greeno 
& Hall, 1997; Mayer, 2001; Ryan, Hodson-Carlton, & Ali, 2005), 

• the findings of the study supported those 
findings, 

• cognitive load was reduced and this 
supported subsequent learner retention. 
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• In addition, the study found that ?, 

the factors of embedded graphics \/ 
and metaphors, followed by the 
integration of visual multimedia, 
such as embedded multi-media 
PowerPoints, or MP4s, was more effective 
than some other multi-media, such as 
Wimba or Second Life to lessen cognitive 
load and best predict student retention for 





Findings 

While newest technologies can be 
used as a competitive tool, there 
may be other factors that could 
provide deep solutions to facilitate 
integration of new technologies 
seamlessly into the fabric of their 
learning and help students weave 
new content knowledge into their 
daily lives. 
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Findings 

• This study was conducted to see how student 
cognitive load could be minimized and 
monitored so that new technologies could be 
integrated seamlessly into learning, instruction 
and performance. 

• The study provided insights into which factors 
could help regulate cognitive load for students. 

• Further studies could provide more data on 
how these factors could reduce student attrition 
in online courses. 




The study 

graphics and metaphors, followed by the 
integration of visual multimedia, such as 
embedded multi-media PowerPoints, or MP4s, 
were more effective than some other multi-media, 
such as Wimba or Second Life, to lessen cognitive 
load and best predict student retention for adult 
learners 
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• New technologies are often implemented by 
colleges as a kind of panacea to stay competitive 
with other universities or as remedies for 
problems such as attrition. These reasons can 
be valid, particularly as a competitive tool, but 

• there may be other factors that could provide 
deep solutions that could both facilitate 
integration of new technologies seamlessly into 
the fabric of their learning, and help students 
weave new content knowledge into their daily 
lives. 




• The students expressed recognition of simple 
signs and icons, and acknowledged that these 
did serve as kind of signposts that continued to 
point them in an onward direction through the 
course. 
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• One of the biggest surprises in their findings was how 
much the students seemed to get engaged in telling their 
own stories. (Narratives were embedded to function as 
yet another form of semiotics). 

• The vast majority of students expressed the most 
satisfaction in reading about and sharing stories and 
experiences with each other. 

• Both researchers were genuinely surprised 
that the students stayed involved and 
a larger number than expected remained in 
the course through 
its 

completion. 









but there may be other factors that could 
provide deep solutions that could both 
facilitate integration of new technologies 
seamlessly into the fabric of their learning, 
and help students 



weave 



new content 



knowledge 
into their 



daily lives. 
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• This study was conducted to see how 
student cognitive load could be minimized 
and monitored so that new technologies 
could be integrated seamlessly into learning, 
instruction and performance. 

• The study provided insights into which 
factors could help regulate cognitive load for 
students. Further studies could provide more 
data on how these factors could reduce 
student attrition in online courses. 



Thank You! 




49 







References 



Ambron, S. & Hooper, K. (Eds.) (1990). Learning with interactive multimedia: 
Developing and using tools in education. Washington: Microsoft Press. 

Baddeley, A.D. (1992). Working memory. Science, 255, 556-559. 

Bailey, R. (1999). Multimedia instruction. Thousand Oaks, CA: Sage Publications. 

Basak, C. (nd). Expansion of focus of attention (FoA)in working memory: Converging 
evidences. Downloaded on January 6, 2008 from 

http://74.125.95. 132 '/search?q=cache:xTvP9hYKLvEI:memorv.svr.edu/courses /Prose 
m Talks /FoA -Liam.ppt+ workine+memorv+expansion&hl=en&ct=clnk&cd=2&pl=us 
Beggs, T. A. (2002). Influences and barriers to the adoption of instructional technology. 
Carrolton, GA: State University of West Georgia [WWW document]. URL 
http://www.mtsu.edu/~itconf/proceedOO/beggs/beggs.htm 
Berker, A. & Horn, L. (2005). Work first, study second: Adult undergraduates who 
combine employment and postsecondary enrollment, The Education Statistics 
Quarterly Volume 5, Issue 3, 2004. 

Birdsong, D. (1999). Second language acquisition and the critical period hypothesis. 
Mahwah, NJ: Lawrence Erlbaum Associates. 

Bradburn, E., Berger, R., Li, X. Peter, K. & Rooney, K. (2004). A descriptive summary of 
1999-2000 bachelor's degree recipients 1 year later: With an analysis of time to degree. 
Education Statistics Quarterly Vol5, Issues 3 & 4. Retrieved December 7, 2006 from 

http://nces.ed.gov/programs/quarterlv/vol 5/5 3/4 3.asp 




Brooks, D. (2008). The cognitive age. TheNewYork Times, May 2, 2008.. Retrieved 
on November 15, 2008, from http://neuroanthropoloev.net/2008/05/03/david- 
brooks-part-one-the-cognitive-age/ 

Cobley, P., Jancz, L. (1997). Introducing semiotics. Royston, United Kingdom: Icon 
Books. 

Cooper, G., (1998). Research into cognitive load theory and instructional design at 
University of New South Wales (UNSW). Retrieved on December 1, 2006 from 
http: / /projects, ict.usc.edu/itw/ materials / clark/UNSW.htm 
Crawford, C., Gannon-Cook, R. (2008, July). Building autonomous and dynamic 
communities of learning within distance learning environments. The International 
Journal of Technology, Knowledge and Society. Available at http: / /iit.cgpublisher.com 
Driscoll, M. (2000). Psychology of learning for instruction. Boston, MA: Allyn and 
Bacon. 

Gannon-Cook, R. (2006 Winter). Crossing virtual borders for interdisciplinary 
research. International Foundation of Teaching Sciences (WOTS). 

Gannon-Cook, R., Crawford, C. (2006). What cave walls can teach us? In A. 
Edmundson (Ed.), Globalized E-Learning Cultural Challenges. Minneapolis, MN: 
eWorld Learning. 

Gannon-Cook, R., Crawford, C. (2007). Persistently simple: Lessons learned from 
research on adult e-Learning courses. Proceedings of the E-Learn International 
Conference, American Association for Computer Education, Quebec, Canada. 

Hard, M. & Jamison, A. 200S.Hubris and hybrids: A cultural history of technology 

and science. Routledge. 



50 






Hudson, L. & Shafer, L. (2005). Undergraduate enrollments in academic, career, 
and vocational education, The Education Statistics QuarteriyVo\_ 6, Issues 1 & 2. 
Retrieved December 7, 2006 from 

http://nces.ed.gov/programs/quarterlv/Vol 6/1 2/5 l.asp 

Huffman, D., Goldberg, F, & Michlin, M. (2003). Using computers to create 
constructivist learning environments: Impact on pedagogy and achievement. 
Journal of Computers in Mathematics and Science Teaching 22(2), 151-168. 
[Online], Available: http://dl.aace.org/13529 

Jewitt, C. & Oyama, R. (2001). Visual meaning: A social semiotic approach. In T. 
vanLeeuwen and C. Jewitt, (Eds.), Handbook of visual analysis (p.134-156) 
October, 1999. Thousand Oaks, CA: Sage Publications. 

Kegan, R., 1994. In over our heads: The mental demands of modern life. 
Cambridge, MA: Harvard University Press.) 

Mayer, R. (2001). Multimedia learning. New York: Cambridge Unversity Press. 
Mayer, R. (2005). The Cambridge handbook of multimedia learning. New York: 
Cambridge University Press, 2005. 

McKenney, S., & van den Akker, J. (2005). Computer-based support for 
curriculum designers: A case of developmental research. Educational 
Technology Research and Development, 53(2), 41-66. 

National Center for Education Statistics (NCES) (2002). A profile of participation 
in distance education: 1999-2000. Retrieved January 15, 2002, from 

http: / / www. nces.ed.gov/pubs02 / 




Nelson, B., Ketelhut, D. J., Clarke, J., Bowman, C., & Dede, C. 
(2005). Design-based research strategies for developing a 
scientific inquiry curriculum in a multi-user virtual 
environment. Educational Technology, 45(1), 21-28. 
Nersessian, N. (2007). Mental modeling in conceptual change. 
In S. Vosniadou (ed.), Handbook of Conceptual Change. 
Mahwah, NJ: Erlbaum Associates. 

Pavio, A. (1990). Mental representations: A dual coding 
approach. New York : Oxford University Press. 

Preskill, H., Catsambas, T. (2006). Reframing evaluation 
through appreciative inquiry. Thousand Oaks, CA: Sage 
Publications. 

Richey, R., and Klein, J. (2007). Design and development 
research. Mahwah, NJ: Lawrence Erlbaum Associates, 
Publishers. 



51 






Ryan, M., Hodson-Carlton, K. & Ali, N.S. (2005). A model for faculty teaching online: 
Confirmation of a dimensional matrix. Journal of Nursing Education, 44(8), 357-365. 
Singleton, R. A, & Straits, B. C. (2005). Approaches to social research (4 th ed.). New 
York: oxford University Press 

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. 
Cognitive Science, 12, 257-285. 

Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. 
Learning and Instruction, 4, 295-312. National Center for Education Statistics (NCES) 
(2002). A profile of participation in distance education: 1999-2000. Retrieved 
January 15, 2002, from http://www.nces.ed.gov/pubs02/ 

Tabachnick, B. & Fidell, L. S. (2001). Using multivariate statistics (Fourth Edition). 
NY: HarperCollins College Publishers. 

Vygotsky. L. (1962). Thought and language. Cambridge, MA: MIT Press. 

Vygotsky, L. (1978). Mind in society. Cambridge, MA: Harvard University Press. 
Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), 
The concept of activity in Social Psychology. Armonk, NY: Sharpe. 

Wertsch, J.V. (1985). Cultural, communication, and cognition: Vygotskian 
perspectives. Cambridge University Press. 



52 




Faculty Training for Online Teachers in Three Rural Alabama Community 

Colleges: A Collective Case Study 

Robert E. Davis Jr., Ph.D. and Angela D. Benson, Ph.D. 

Abstract 

Online education is providing community colleges opportunities to reach students through the use of technological 
advances. These technological advances have allowed faculty to develop new teaching skills and transfer these skills 
into pedagogical practices in an online environment. The purpose of this study was to examine how the training 
instructors receive support their teaching classes in an online environment at community colleges. This research 
study followed a qualitative research design; a collective case study that included administrators who made the 
decisions for online education, trainers who provided professional development training, and faculty who had taught 
online classes at community colleges. In response to the overarching question that guided this study, “How does 
professional development training influence faculty teaching online at Alabama’s rural community colleges?” this 
study found that technological skills and familiarity with online education theories and pedagogies facilitated the 
transfer of pedagogical strategies from the training to their online classes. The study found that when faculty 
received training incorporating online pedagogical strategies, the online teaching practices taught during the training 
were transferred to their online classes. This study also made clear that the instructors’ interest in online education 
was a driving force for the establishment of formal professional development training programs at their institutions. 
The results of this study can inform all community colleges of the need for professional development training, spur 
innovations in faculty training for online teaching, and encourage further research in the area of training for online 
teaching and learning. 

Introduction 

Professional development for community college faculty began to grow in response to the realization that 
the rapid growth of new community colleges in the early 1960s and 1970s was waning and that people, rather than 
buildings, programs, and organizational structures, needed attention (O’Banion, 1981; Watts & Hammons, 2002). 
Hammonds, Wallace, and Watts (1978) identified multiple events that led to professional development at 
community colleges, including (a) the need for increased effectiveness and efficiency due to competition for limited 
tax dollars and beginning public demands for accountability, (b) the acknowledgement that the future success of the 
community college depended on the ability of its personnel to adapt to a constantly changing environment, (c) the 
development of a technology of instruction with potential for improved instruction unknown to most faculty, (d) an 
awareness to faculty that they were becoming unable to cope with the needs of the increasing percentages of “high 
risk” students enrolling in community colleges, (e) a recognition among leaders that change was imperative and they 
needed to become skilled in planning, implementing, and evaluating change, (f) the increasing influence of court 
decisions, collective bargaining, and federal regulations on instructional governance, and (g) the occurrence of a 
relatively high turnover in leadership positions at the midmanagement levels. 

Community colleges are continuing to change in response to community and societal changes, and those 
who lead, teach, and provide support in those colleges will need to continually grow and change as well (Watts & 
Hammons, 2002). According to Watts and Hammons (2002), “Professional development has provided and will 
continue to provide the necessary programs to meet growth needs and appears now to be a permanent fixture in 
community colleges.” (p. 8) 

The adoption of online education practices has produced important changes in education in the last decade, 
including a growing tendency towards new models of teaching and learning that include a combination of web- 
based technologies and classroom instruction (Bonk, Kim, & Zeng, 2006; Dziuban, Moskal, & Hartman, 2004; 
Horizon Report, 2005). This use of web-based systems has led to an increase in the offering of distance education 
courses by higher education institutions (Lewis, Snow, & Farris, 1999), and has had an impact on classroom 
education as well (Bates & Poole, 2003; Curtain, 2002; O’Sullivan, 2000; Palloff & Pratt, 2001; Spector & Teja, 
2001 ). 

Successful online course experiences for students and instructors depend on the expertise and dedication of 
a well-prepared online instructor (Ko & Rossen, 1998). New instructors need comprehensive training to ensure a 
strong start, and continuing support and services throughout their distance education experience to promote 
maximum quality and satisfaction in the online courses they teach (Lieberman & McNett, 2000). When online 
instructors are fully equipped with proper technical skills, familiar with the online learning environment, and most 
importantly, hold a true awareness of effective online pedagogies and teaching strategies, the online learning 
experiences they create have the highest potential to succeed (Bedore, 1997). 
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Purpose 

Given the evidence that the use of technology provides new opportunities for learning, for collaboration, 
and for developing learning communities (Cross, 2003; Horizon Report, 2005), there is an impetus for community 
colleges to encourage faculty to integrate technology in their teaching and learning, The purpose of this study was to 
examine the training that instructors received in support of their online classes at the community college and to 
contribute research on professional development training. The results of this study can be used to enhance 
professional development programs, to develop new strategies for faculty training, and to equip instructors with 
skills that support pedagogical decisions when integrating new approaches, which include technology. 

Theoretical Framework 

Transfer of learning is the degree to which an individual transfers the knowledge, skills, and attitudes 
gained formally or informally to new situations (Baldwin & Ford, 1988; Enos, Kehrhahn, & Bell, 2003). Transfer is 
a critical element in learning, and it involves complex cognitive processes. Transfer occurs when an individual is 
faced with a situation or problem and draws from previously acquired knowledge to deal with this new circumstance 
(Ormrod, 2004; Schunk, 2004). 

Training, including professional development training, is useless if it cannot be transferred into 
performance (Yamnill & McLean, 2001). Training is of little value to organizations (and community colleges) 
unless it is transferred in some way to performance (Holton, Bates, Seyer, & Carvalho, 1997). Therefore this study 
used Holton’s (1996) transfer of training model as a guiding framework. This model proposes three primary 
outcomes and three primary inputs of a training intervention. The three primary outputs are: learning, individual 
performance, and organizational results. The three primaiy inputs are: motivation to transfer, transfer climate, and 
transfer design (Holton, 1996). 

According to Holton (1996), motivation to transfer comprises four categories: (a) intervention fulfillment 
(e.g., refers to the perception of learning something that has met the trainee’s expectations and has fulfilled his/her 
need for performance-related learning), (b) learning outcomes (e.g., the results of learning will have an effect on 
motivation to transfer, this is also related to the expectancy theory), (c) job attitudes (e.g., meaning that a person’s 
commitment and job satisfaction should influence motivation to learn and to transfer to the job), and finally, (d) 
payoff (e. g., one who sees utility in training is more motivated to transfer to the job than one who does not). 

Holton’s (1996) transfer of training model included “transfer climate” which is defined as the mediating 
variable between organizational context and a person’s job attitudes and work behavior as a result of an individual’s 
perception of their work environment. Thus, even when learning occurs in training, the transfer climate may either 
support or inhibit the application of learning to the job or the extent to which a person can use learned skills on the 
job (Mathieu, Tannenbaum, & Salas, 1992). 

Holton’s (1996) definition of training design can be characterized as the conditions necessary within a 
training program for facilitating the transfer of training. Training design can be summarized as the characteristics of 
the learning environment, such as the materials, opportunities to practice, providing feedback, development of 
learning objectives, and organizational results. 

Holton (1996) provided a definition of learning as the achievement of the learning outcome desired in an 
HRD intervention. For this study “an HRD intervention” means a training program. 

Holton (1996) defined this as a change in individual performance as a result of learning being applied on 

the job. 

Holton (1996) defined organizational results as results at the organizational level as a consequence of 
change in individual performance. Simmons (1999) further expanded organizational results as special measures of 
the organization that can be taken before and after learning takes place and to help the process of transfer to the 
workplace. 

Research Questions 

The overarching question, which this research addresses, is “How does professional development training 
influence faculty teaching online at Alabama’s rural community colleges?” To guide the data collection and 
analysis, the following sub questions were explored: 

1. What professional development training (including follow-up training) is provided to community college teachers 
preparing to teach online? 

2. How do the elements of Holton’s Transfer Model influence the application of this professional development 
training to teaching the online class? 
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Methods 

This study employed a qualitative case study research design; more specifically, the study was designed as 
a collective case study (Stake, 1995). Case studies are intensive descriptions and analyses of a single unit or 
bounded system (Merriam, 1998). The bounded system in this case study is the community college with the unit of 
analysis being the community college staff involved in the delivery of online courses, including instructors, trainers 
and distance learning coordinators. 

Selection of Sites and Participants 

Community college criteria and selection. Three Alabama community colleges were selected: Alpha 
Community College, Beta Community College, and Charlie Community College. The following criteria were used 
in selecting the community colleges: 

1 . Offer online learning courses, 

2. Distance Learning Coordinator responded to initial email questionnaire, 

3. Used fulltime faculty and staff to conduct professional development training, 

4. Fulltime faculty who teach both traditional and online courses, 

5. Rural Alabama college classification by The Carnegie Foundation for the Advancement of Teaching, 

6. Classification as Associate level institutions where all degrees are at the Associate level, and 

7. One college each from the classification of small, medium, and large populations based on The 
Carnegie Foundation for the Advancement of Teaching. 

Participant criteria and selection. The researcher selected fourteen participants who were involved in 
online education delivery including: distance learning coordinator (DLC), trainers (T), and instructors (I) from the 
three community colleges. The selection criteria used were the following: 

• Distance Learning Coordinators: 1) Designated as the distance learning coordinator at the community 
college, and 2) Job title or duties involve distance learning. 

• Trainers: 1) Faculty or staff members of the community college, 2) Conducts training sessions at the 
college, 3) Job title or duties involve training faculty members, and 4) Referred to researcher by the 
distance learning coordinators. 

• Instructors: 1) Full-time instructors, 2) Experienced in teaching both traditional and online courses, 3) 
Taught online classes during the fall semester 2008, and 4) Referred to researcher by the distance learning 
coordinators. 

Data Collection 

Data were collected via in-depth semi-structured interviews that were informal and interactive with open- 
ended questions (Merriam, 1998). The researcher took detailed notes and audio taped each interview. After the 
interview, each recorded interview was transcribed verbatim. 

This study included multiple sources and multiple methods of data collection (Merriam, 1998; Miles & 
Huberman, 1994; Stake, 2000). The methods involved interviews, observations, and study of documents. 

Specifically, four steps were followed in the data collection process: (a) pilot study in which distance learning 
coordinators were interviewed, (b) open-ended interviews with instructors and trainers involved in online learning, 
(c) in-depth study of documents relating to professional development offerings at the participating colleges as well 
as online teaching materials, (d) summary write-up of each case identifying themes related to research questions, 
and (e) cross-case analysis of the individual case summaries. 

Limitations 

Unlike quantitative research studies, the design of qualitative research lacks a consistency, or uniformity, of 
a prescribed procedure. Therefore, this study was limited in the following ways: 

1 . This study was restricted to participants teaching online classes in rural Alabama Community Colleges. 

2. Due to the sample size and absence of randomization, the sample did not allow for statistical 
generalizations. 

3. The Distance Learning Coordinator at each college selected the participants based on criteria developed 
by the researcher. Given this, the selection of participants could be biased. 

Additionally, the qualitative data gathered by the researcher may be subject to other interpretations thereby 
reducing the degree of external reliability of the results. Therefore, generalizing the results to other community 
colleges outside the State of Alabama may not be possible. 

Findings 

Professional Development Training 

All three of the colleges included in the study had established comprehensive professional development 
training programs at the time of review. There were similarities and differences in the manner in which professional 
development training was developed and provided to online instructors. 
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Similarities. The training programs evolved from the instructors who were teaching online classes from the 
initial startup phase in 2001. These instructors were selected on a volunteer basis and basically had no training 
provided to them. They learned how to teach online classes through on-the-job training or by “trial and error.” Since 
training had not been provided these instructors formed informal committees where they would meet and provide 
each other with successes and failures of online education they had learned from experience. At all three colleges the 
informal committees were recognized by administrators, which eventually led to the professional development 
training programs being established for distance education. Another avenue many of the instructors utilized for 
training purposes were their participation in graduate courses. Instructors were pursuing advanced degrees from 
universities, which were tailored to online education. In fact, many of the classes they had to complete for these 
degrees were taught in an online environment. 

Participants at Charlie Community College suggested there had been external influences on decisions made 
at their college. Charlie was in the process of updating their learning management system when a State-level 
committee was formed to standardize the learning management system for all community colleges. However, this 
committee has been inactivated, thereby placing Charlie Community College in limbo as to which direction the 
college should proceed in obtaining a learning management system. Even though this influence was not mentioned 
at the other two colleges, they are part of the same organization, the Alabama Community College System, so the 
same observations apply to them as well. 

Online instructors at all three colleges participating in the study understood that distance education was a 
new shift in teaching methods based on economic factors and that students wanted this type of learning. Therefore, 
each instructor volunteered to teach online classes and found creative ways to obtain the training they needed to do 
so. 

Differences. Alpha State utilized contracted Blackboard personnel for their professional development 
training, which was provided once a year during the fall semester. At the time of this study, this was different from 
Beta and Charlie State, in that neither of these schools had established Divisions within their colleges that provided 
training sessions year round. 

Also, there were differences in the manner in which follow-up training was provided. While Alpha State 
had two instructional designers who provided the majority of follow-up training in one-to-one mode, Beta and 
Charlie State incorporated the professional development training sessions into their training websites. These 
websites allowed instructors to refresh their training knowledge by accessing the training site anytime they needed. 
Of course, there were also personnel within the training divisions who would provide face-to-face help to any 
instructor that requested assistance. 

Another difference in the training programs was that Beta State required all instructors to participate in 
professional development training for distance education whether they taught online classes or not. This requirement 
was to allow all instructors to become familiar with the online process. Further, Alpha State required all division 
chairs and associate deans to attend distance education professional development training also. 

Transfer of training 

The Holton Transfer of Training Model captured the professional development of the community college 
participants in this study. This model proposed three primary inputs: motivation to transfer, transfer conditions 
(environment), and transfer design and three primary outcomes of training interventions: learning, individual 
performance, and organizational results (Holton, 1996). 

Motivation. Motivation to transfer is defined as an individuals’ desire to use knowledge and skills on-the- 
job. Instructors were motivated by training they received on the pedagogy and the valuable practical experience of 
having been an online student while enrolled in graduate school. At all three colleges, instructors who had taken 
graduate courses concentrating on how to teach in an online environment understood online education was changing 
the role of teaching and that it was very important for them to adapt to this change. Because of this knowledge, 
instructors teaching online at Alpha motivated newer online instructors by conducting small workshops on the 
methodology of teaching in an online environment. At Beta Community College instructors were given the 
opportunity to use “virtual hours” to teach their online classes from home. Another motivating factor prevalent at all 
three colleges was the use of a cohort system where instructors teaching online classes would informally meet and 
share their thoughts and ideas on how to better their online classes. 

Transfer design. Transfer design is defined as the conditions necessary within a training program for 
facilitating the transfer of training. All three colleges provided training to their online instructors in a real life work 
environment. All the colleges provided training in a face-to-face environment. Beta and Charlie went one step 
further by recording and posting face-to-face training sessions to their training website. Factors in the training design 
that allowed transfer of knowledge were administration support of online classes and the use of instructional 
designers as training personnel. The workplace atmosphere was an environment that allowed faculty to maintain a 
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positive attitude towards online education even when training on this subject was not available. Because of this 
attitude teachers at all three colleges volunteered to teach online classes, developed and delivered small workshops 
to help and encourage faculty teaching online, and conducted informal meetings to discuss ideas on ways to better 
their online classes. 

Transfer climate. Transfer climate is defined as the mediating variable between organizational context and 
a person’s job attitudes and work behavior as a result of an individual’s perception of their work environment. The 
colleges provided online instructors with a positive climate for transferring their knowledge to their online classes. 
All three colleges provided instructors a web-based system for use when teaching both their online and traditional 
classroom. All three colleges had established formal Distance Education Committees and had strong administration 
support. The committees, with the support of administration, developed a vision for distance education at their 
colleges. In addition, at Alpha instructional designers had been appointed, while Beta and Charlie had incorporated 
training sessions into a training website. This website allowed new instructors the opportunity to participate in the 
previous training sessions and allowed present instructors the opportunity for refresher training. 

Learning and Individual Performance. Learning is defined as the desired outcome of a training program. 
Individual performance is defined as a change in individual performance as a result of learning being applied on the 
job. There were several factors present that contributed to administrations desired learning goal for online teachers 
to strengthen their online classes through training focused on distance education. Instructors at all three colleges 
converted their traditional classroom into a format to be taught in an online environment. One of the tools the online 
instructor’s utilized at all three colleges, which they learned about during professional development training, was the 
discussion board. All instructors incorporated student discussions when introducing course materials in the online 
class, used a questions forum and email to address students ongoing questions, and gave tests to determine students’ 
understanding of the course materials. Alpha established testing centers where all online students were required to 
take their tests, Beta and Charlie instructors gave timed tests, but Charlie instructors require students to take finals 
on campus. 

Organizational results. Organizational results are defined as changes that occur at the organizational level 
as a consequence of a change in individuals. This study did not explicitly explore organizational results but some 
overall observations can be made. Initially there was no training offered when online classes began at the colleges so 
one organizational change outcome was the formation of some form of training to guide and direct teachers toward 
when teaching online. At one college, training was initiated in the form of Blackboard training sessions provided via 
contractual agreements with Blackboard personnel. Also, this college required all division chairs and associate deans 
to attend this training. All the colleges in the study had developed policies and procedures for the distance education 
programs that they provided in a Distance Education Faculty Manual. In addition, technology offices were 
established at the colleges and staffed with qualified individuals in charge of conducting training sessions on 
distance education matters. All three colleges provided face-to-face training sessions and at two of the colleges, the 
training material used was posted on their college online education websites. 

Discussion 

Although previous studies have suggested that professional development training provide pedagogical 
strategies for online teaching in the classroom, there were no studies, which explored this topic in Alabama 
Community Colleges. The results of this study were consistent with previous researchers’ suggestions (Henning, 
2000; Simpson & Head, 2000) indicating that professional development training was a major issue on faculty’s 
ability to design and teach online courses. 

This study identified several factors that contribute to professional development training that help faculty 
teach online. These factors include: administration support (transfer climate), technology staff support (transfer 
climate, transfer design), training curriculum (transfer design), and motivation to transfer. 

Administration Support (Transfer Climate) 

The administration’s support is a major key to offering professional development training to online 
teachers. Even though, as this study showed, teachers will obtain training through other avenues, it is essential to 
provide training through the college. The attitudes of the teachers toward online teaching are directly tied to how 
they perceive they are supported by administration. Administration was supportive at all three colleges. In fact, 
training would not have been established at the three colleges had it not been for the support of administration. 
Technology Staff Support (Transfer Climate, Transfer Design) 

The role of the technology staff is another major key to offering professional development training to 
online teachers. The colleges included in the study had well-organized technology staffs eager to help instructors 
increase their knowledge of online teaching strategies. These individuals developed and provided training to all 
instructors at the colleges and provided software suggestions to enable instructors to enhance their online classes. 
Training Curriculum (Transfer Design) 
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All three colleges had spent countless planning hours configuring their training program for online 
instructors. Alpha State utilized yearly training sessions and instructional designers to teach the proper techniques of 
online teaching. Beta Community College and Charlie Community College utilized multiple training sessions 
offered each semester (summer excluded) in computer labs. In addition they captured each training session and 
posted them to the college websites. Doing this provided instructors a mechanism to review the sessions whenever 
they felt they needed follow-up training. Directors of Training at Beta and Charlie Community College 
corresponded with each other on a regular basis to share successes and hardships and learn from each others’ 
programs. 

Motivation to Transfer 

All the participants volunteered to teach online classes and came to understand the importance of the online 
education program at their college. They understood this was a new form of education desired by students and 
wanted to become a part of the first to teach online classes at their college. Faculty also sought training experiences 
when their colleges offered none. 

Summary 

This study found that faculties were not provided training during the initial start-up phase of online 
education because the community college administrators assumed teachers would know how to teach in the new 
environment. They soon found out that training and assistance were much desired by their online instructors. This 
study suggested that since their institution did not offer training, these faculty members obtained training through 
other means. As a result these faculty members were instrumental in helping the institutions establish formal training 
programs. 

Additionally, this study, like other recent research on training (Fenby, 2006), suggested training should take 
place in an online learning environment to give the faculty training in the pedagogy and experience of having been 
an online learner. This study suggested training provided faculty with the readiness to teach online classes. The 
training provided familiarity with technology and with the pedagogy of using course content and context in an 
online environment. 

Implications for Practice 

The study identified factors that facilitated the effect professional development training had on faculty at 
Alabama’s rural community colleges in the development of online classes. These findings also have implications for 
the practice for community colleges offering or planning to offer online classes. 

Faculty Professional Development Training Curriculum 

Faculty training for online teaching should include modules on pedagogical strategies in addition to 
modules on the mechanics and application of online tools. Training which was focused entirely on the mechanics of 
the application of software tools and did not provide information on the pedagogical strategies fails to provide 
faculty with the knowledge needed to teach online (Palloff and Pratt, 1999). Consistent with previous research, this 
study found that online teaching should focus on pedagogy (Bates, 2000; Bates & Poole, 2003; Boettcher & Conrad, 
1999; Palloff & Pratt, 2001; Spector & Teja, 2001) and how technologies can provide alternate ways of knowledge 
and uses of information as they taught and prepared their first online class (Fenby, 2006). 

Follow-Up Training/Instruction 

Follow-up instruction should be provided to increase the likelihood that faculty will apply what they learn 
during training in their online class. Training that does not allow for follow-up instruction does not provide faculty a 
mechanism to refresh and reiterate the pedagogical strategies learned in the initial training. Follow-up instruction 
activities identified in this study included: (a) training captured by video or audio taped and put on the college 
website, and (b) one on one sessions by the instructional designers, as needed. 

Support System For Faculty Teaching Online 

Establish a community of online teachers — whether face-to-face or online. Faculty members who had 
taught online classes after participating in professional development training for online teaching needed to be given 
opportunities to reflect on their online teaching experiences. This helped them produce changes in their teaching 
methods. Informal learning from their colleagues, as this study showed, played an important role as a motivator for 
reflection and exploration of new ideas. The colleges studied provided avenues to facilitate and encourage 
opportunities for faculty reflection and sharing of teaching methods. These initiatives enhanced transfer and 
ultimately increased faculty interest in integrating online technologies. 
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Training for All 

Require all instructors not teaching online classes to have exposure to the online software and to use online 
tools for some aspect of their class. Also, administrators should be required to attend this training to ensure they 
have first-hand knowledge of the online classroom experience. This knowledge will allow administrators who are 
appointed to state-level or local level committees dealing with online teaching to have some personal knowledge of 
the subject. 

Staffing of Training Effort 

Require training personnel to have exposure in either actual online teaching or extensive professional 
development training in the proper techniques of teaching in an online environment. The training staff should have 
first-hand knowledge of teaching in an online environment. Also, require the use of instructional designers to 
strengthen training programs. These individuals should have prior experience teaching in an online environment or 
have had extensive training in the area of online education. Administrators should consider appointing instructional 
designers from the pool of instructors who are teaching online classes at their college since they would have the 
necessary experience and also be familiar to the faculty. 

Organizational Structure for Distance Education 

Administrators should strongly consider establishing the training programs for online education as a 
separate division of the college with a separate Dean, Director of Technology, Distance Learning Coordinator, and 
training personnel. This Division should maintain a separate funding line to ensure accountability of funding and 
student learning outcomes. Since teaching in an online environment requires additional and special technological 
resources a separate division would make the program more managable. Also, this will broadcast the college’s 
commitment to online distance education and allow the college to maintain a separate strategic plan, resulting in a 
stronger distance education program. 

Delivery of Training 

Provide training in an environment that realistically represents the online classes at the colleges. One 
recommendation is to establish training centers on campus where not only faculty, but students, can receive training 
on the proper techniques for teaching and learning of an online class. Training personnel should deliver face-to-face 
instruction to faculty members and provide follow-up training via the web or instructional designers. Capturing the 
actual face-to-face training and posting it to a training website where instructors can access anytime would provide a 
tremendous amount of training hours to faculty. Establish a separate instructor and student training module for the 
delivery of follow-up training. 

Vision Sharing 

Reinstate or establish State -level committees to provide oversight and vision sharing to all community 
colleges within the State. This committee would provide an avenue for sharing distance education procedures among 
community colleges. Ensure the committee members selected to represent the respective community college have 
first-hand knowledge on teaching in an online environment. 

Reflective Commentary 

As the researcher, I was surprised to find the three colleges reviewed had established comprehensive 
professional development training programs. Based on my personal experience in the education field I had initially 
thought there would be some training provided for distance education instructors but not to the extent that these 
findings revealed. It was exciting to see in practice things discussed while taking instructional technology courses. 
For example, one class I completed had revealed the importance of instructional designers in developing and 
sustaining online classes. I was actually permitted to see this in action when I collected data from Alpha State. This 
reinforced the importance of the role instructional designers have on training curriculum for online instructors. 
Another key training ingredient that I enjoyed was the training websites developed at Beta and Charlie State. This 
reinforced my understanding of the importance of internet websites in providing instructors with immediate training 
or in other words, training “just in time.” 

Recommendations for Further Research 

There are still elements that remain unexplored dealing with professional development training in 
Alabama’s rural community colleges. Some items that need further study are: 

1 . Include a larger number of participants to lend additional validity to the initial findings of this 
research. 

a. Include adjunct faculty who teach online classes. 

b. Include faculty who have not taught online classes. 

c. Include additional community colleges. 

d. Compare community colleges within an urban/suburban and/or another sector. 
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Include participants at the State-level to determine: 

a. What policies and procedures are being reviewed and/or developed to standardize online 
education within The Alabama Community College System. 

b. Whether state-wide committees of distance education should be reactivated and include 
representatives from colleges who have experience in an online environment. 

3. Perform a quantitative approach and use a statistical survey across all community colleges. 

Conclusion 

In response to the overarching question that guided this study, “ How does professional development 
training influence faculty teaching online at Alabama’ s rural community colleges?” this study found that 
technological skills and familiarity with online education theories and pedagogies facilitated the transfer of 
pedagogical strategies from the training to their online classes. The study found that when faculty received training 
incorporating online pedagogical strategies, the online teaching practices taught during the training were transferred 
to their online classes. This study also made clear that the instructors’ interest in online education was a driving 
force for the establishment of formal professional development training programs at their institution. Results from 
this study have the potential to inform innovations in faculty training for online teaching and to encourage further 
research in this growing area. 
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Abstract: Variations in the policies used by virtual schools in relation to course 
enrollment trial periods and course completion impact the comparability of attrition 
statistics. Based on the responses of 86 of the 159 U.S. virtual schools we contacted, 
approximately 70% had trial periods that varied from one day to 185 days. Course 
completion definitions also varied considerably from remaining in the course 
irrespective of the final grade to receiving an A- in the course. We recommend 
virtual schools adopt consistent measures for calculating student attrition to allow 
meaningful comparisons between virtual schools and also between virtual schools 
and brick-and-mortar schools. 

There has been a rapid proliferation of virtual schools throughout the United States. In 2008, 44 states had 
significant K-12 online activity (Watson, Gemin, and Ryan 2008), up from 21 states just three years earlier (Watson 
and Kalmon 2005). One challenge to such rapid growth has been adopting common metrics across virtual schools to 
measure quality. Currently, no common metrics exist to calculate course completion rates (Pape, Revenaugh, 
Watson, and Wicks 2006; Smith, Clark, and Blomeyer 2005, Watson and Ryan 2006). Unfortunately, completion 
rates are a key indicator commonly used to measure the quality of virtual schools (Blomeyer and Dawson 2005). 

The absence of a common metric for measuring course completion rates makes it difficult to draw comparisons on 
student retention and attrition among virtual schools. 

Several factors affect course completion rates. While many are directly related to the student and the 
learning environment (Rice 2006), administrative policies unique to virtual schools can also affect course 
completion rates. Policies that determine when and how students are counted in the official course completion rates 
can affect the outcome measure (Roblyer 2006). The purpose of this study was to explore the policy variations in 
how U.S. virtual schools calculated completion rates. Specifically, this study sought to answer the following 
research questions. 

1. What are the trial period policies for US virtual schools? 

a. How much variability exists across different types of virtual schools and regions of the country? 

2. How do US virtual schools define course completions? 

a. How much variability exists across different types of virtual schools and regions of the country? 

Review of Literature 

In virtual school settings, student attrition is believed to be a significant problem (Carr 2000; Rice 2006). It 
is estimated that attrition rates range between 12%-40% (Lary 2002) and in some cases may even be as high as 50% 
(Rice 2006). These estimates vary greatly due to differences in the age of the program, method and amount of 
funding, instructional quality, teacher staffing, technical support, and the diversity of the student body (Roblyer 
2006). Another contributing factor is the lack of a commonly accepted metric for calculating student attrition. While 
most virtual schools calculate student attrition, no common criteria exist on when a student is officially enrolled and 
what constitutes a course completion (Pape et al. 2006; Smith et al. 2005). Simply put, a student counted as 
completing a course in one virtual school may be considered a non-completer in another (Watson and Ryan 2006). 
Varying metrics make it impossible to compare attrition rates between virtual schools, and also make it difficult to 
compare attrition rates between virtual schools and brick-and-mortar schools. 

While attrition rates may be high, the seriousness of the problem might be masked by the policies adopted 
to determine when and how students are counted. Many schools utilize a trial period wherein students can withdraw 
from a course with no penalty and not be counted as officially being enrolled (Watson et al. 2008). Long trial 
periods can act as a sifting mechanism when weaker students drop out, masking attrition rates for lower performing 
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students (Barbour and Reeves 2009; McLeod, Hughes, Brown, Choi, and Maeda 2005). Virtual schools with 
generous trial periods would be able to report high retention rates because students who were struggling would have 
dropped out before the virtual school began counting them, in their 1999-2000 evaluation, Bigbie and McCarroll 
(2000) reported that FLVS had a 73.6% completion rate. However when taking into account students who dropped 
out during the 28-day trial period, the completion rate fell to 53.5%. This problem has also been well documented in 
comparative studies examining student performance of virtual school students compared to brick-and-mortar 
students (Balias and Belyk 2000; Cavanaugh, Gillan, Bosnick, Hess, and Scott 2005). Retention figures across 
virtual schools vary since institutions begin counting students at different points in time and this prevents 
meaningful comparisons between virtual schools and brick-and-mortar schools. For example, in their study 
examining students’ algebraic understanding McLeod et al. (2005) found that virtual school students performed 
better than their classroom counterparts, but speculated the reason was due to the high dropout rate in virtual school 
courses. They believed the result of the attrition was that many low-achieving virtual school students had already 
removed themselves from the sample prior to the assessment. This is only one of many possible examples. 

Similarly, the definition of a course completion can also affect retention rates (Pape et al. 2006; Roblyer 
2006). A school that defines course completion as a student receiving a grade (i.e., A-F) will have a higher retention 
rate than a school that defines a course completion as a student receiving a passing grade (i.e., D- or better). Again, 
these variations make it difficult to compare retention rates and weaken the value of the outcome metric. 

While numerous researchers have called for standardizing performance measures across virtual schools 
(Smith et al. 2005; Pape et al. 2006; Watson and Ryan 2006, Watson et al. 2008), there is limited research 
examining the extent of variation that exists in trial period policies and course completion definitions. Pape et al. 
(2006) conducted a small study comparing several outcome measures used at three virtual schools: Virtual High 
School Global Consortium, Illinois Virtual High School, and Connections Academy. Two of the three schools had 
trial periods of three and five weeks. There were also differences in course completion definitions across the 
schools. Watson and Ryan (2007) also surveyed 32 virtual schools to analyze trial period and course completion 
policies to determine how completion rates were calculated among virtual schools. They found that about two-thirds 
of the schools had trial periods of varying lengths and the same amount required students to receive a passing grade 
in order to be counted as a course completion. A major limitation of both Pape et al. (2006) and Watson and Ryan’s 
(2007) was the sample size of 3 and 32 respectively. In addition, neither study examined whether there was a 
relationship between the policies adopted and the geographical region or type of virtual school. Beyond these two 
studies, we were unable to locate other studies examining trial period policies and course completion definitions 
across US virtual schools. This study sought to fill this gap in this body of research on K-12 online learning. 

Methodology 

We surveyed schools using a combination of methods including e-mail, fax, and telephone (see Appendix 
A for a copy of this survey). To guide our study, we used Clark’s (2001) definition of a virtual school as “a state 
approved and/or regionally accredited school that offers secondary credit courses through distance learning methods 
that include Internet-based delivery” (p .i). Schools included in the sample were located using a variety of sources, 
as no comprehensive list existed. We included all of the state-led schools listed in the Keeping Pace with K-12 
Online Learning: A Review of State Level Policy and Practice report (Watson and Ryan 2007) and those listed on 
North America Council for Online Learning’s (NACOL) Online Learning National Clearinghouse for Online 
Programs (see http://www.edgatewav.net/cs/nacol/print/docs/437 ) as of November 2007. The accuracy of 
NACOL’s list was questionable, as we found contact information dated, listings for schools that had closed, and 
course content vendors listed as schools. The combined lists yielded 159 schools, all of which met Clark’s (2001) 
definition of a virtual school. Virtual schools were categorized using the taxonomy outlined in Cavanaugh, Barbour, 
and Clark (2008) and grouped in the geographic regions listed in Watson and Ryan’s (2007). Table 1 describes the 
schools sampled based on school type and geographical region. 
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Table 1. Demographic characteristics of schools sampled 



Characteristics 


Sample Size 


% of Sample 


School type 


Cyber Charter 


34 


21.1 


For Profit 


9 


5.6 


Multi-district / Consortium 


11 


6.8 


Private 


21 


13.0 


Single-district 


49 


30.4 


State - led 


24 


14.9 


University - led 


11 


6.8 


Total 


159 


100 


Region 


Central States 


41 


25.5 


Northeastern States 


18 


11.2 


Southeastern States 


33 


20.5 


Western States 


67 


41.6 


Total 


159 


100 



Single-district and cyber charter schools accounted for over half of the schools sampled compared to for-profit 
schools that accounted for the smallest portion of the sample. Although Northeastern states had a high number of 
students and brick and mortar schools, they had fewer virtual schools and online learning programs compared to 
other regions in the U.S.. Many states in the Northeastern region participated in Virtual High School, a large virtual 
school consortium, which draws students from 500 high schools in 28 states and 35 countries (Watson et al. 2008). 

In Massachusetts, for example, 100 high schools (30%) had students attending the Virtual High School. Moreover, it 
was also not surprising that the bulk of virtual schools sampled were from Western states, which have larger rural 
populations. Traditionally, distance education was used by rural school districts as a means to provide students with 
access to college preparatory courses and qualified teachers otherwise inaccessible (Smith et al. 2005). Thus, the 
sample was reasonably representative of the K-12 online learning landscape. 

Our data collection methods consisted of surveys that were sent initially by electronic mail on two 
occasions, then once by fax, and finally once by phone. We analyzed the data using SPSS statistical software. Trial 
period lengths were measured by days. Data for course completion policies were collected qualitatively. We 
analyzed the responses and clustered course completion statements into three broad themes. Within each of these 
themes there was further differentiation, which we maintained in our quantitative coding. Data were examined using 
basic descriptive statistics, one-way ANOVAs, and cross tabulations. There were some limitations to the nature of 
the data we collected. Six schools identified trial period lengths that varied based on the length of the semester. 
Summer semesters typically had shorter trial periods than winter and spring semesters. In these cases, we coded the 
schools with the longer trial period of winter and spring semesters. Finally, one school reported it had an 
“indefinite” trial period. In this instance, we used the length of the school year for brick-and-mortar schools in the 
same school district to determine the trial period length. 
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Results and Discussion 



Of the 159 sampled, 86 schools responded to the survey. The response rate was 54.1% and varied based on 
school type and region. Table 2 indicates respondents based on school type and geographical location. 

Table 2. Frequency count of respondents based on school type and geographical region 



Demographic Characteristics 


Respondents 


% of Respondents 


School type 


Cyber Charter 


16 


18.6 


For Profit 


1 


1.2 


Multi District / Consortium 


7 


8.1 


Private 


12 


14.0 


Single District 


26 


30.2 


State - led 


16 


18.6 


University - led 


8 


9.3 


Total 


86 


100 


Region 


Central States 


23 


26.7 


Northeastern States 


8 


9.3 


Southeastern States 


19 


22.1 


Western States 


36 


41.9 


Total 


86 


100 



Single district virtual schools had the highest response rate and accounted for one third of the respondents. Western 
state schools accounted for almost 42% of the respondents, as was expected. While Central and Southeastern states 
were nearly evenly split, accounting for approximately half of the respondents. Northeastern states had a lower than 
expected response rate. This was likely due to the fact that so many schools and districts in this region were 
members of Virtual High School. 



To determine the representativeness of the respondents, Table 3 details the proportion of actual respondents, 
proportion in the original sample, and the difference between these two proportions by geographical region. 

Table 3. Regional differences in respondents and total sample 



Region 


% of Actual Respondents 


% Sampled 


% Difference 


Central States 


26.7 


25.5 


1.2 


Northeastern States 


9.3 


11.2 


-1.9 


Southeastern States 


22.1 


20.5 


1.6 


Western States 


41.9 


41.6 


.3 



The difference between the two proportions was small. Consequently, the study’s findings were regionally 
representative of the K-12 online learning landscape in the U.S.. 

Question 1: Trial period policies 

Trial periods were a fairly common practice, with 68.6% of schools having a policy. This finding was 
supported by a much smaller study surveying 32 virtual schools, which found that about two thirds of the programs 
surveyed had trial period policies (Watson and Ryan 2007). There were subtle variations in the prevalence of trial 
period policies based on the type of virtual school and geographical location. For example, all but one state-led 
school had a trial period of some length. Private and cyber charter schools were almost evenly split on the presence 
or absence of trial periods. Regionally, Northeastern states were least likely to have trial periods with only 28.6% 
compared to Southeastern states where 83.3% had trial period policies. Approximately 70% of Western and Central 
States had trial period policies. 

There was significant variation in the lengths of trial period policies reported. Trial period length ranged 
from as little as one day to as long as 185 days. The mean number of days was 25.22 days, while the median was 15 
and the mode was 14. Table 4 reports the frequency counts of the different trial period lengths. 



65 




Table 4. Frequency of trial period length by days 



Days 


n 


% 


1 


2 


3.4 


3 


1 


1.7 


7 


5 


8.6 


8 


1 


1.7 


10 


4 


6.9 


14 


12 


20.7 


15 


4 


6.9 


20 


1 


1.7 


21 


6 


10.3 


28 


7 


12.1 


30 


9 


15.5 


35 


2 


3.4 


60 


1 


1.7 


180 


1 


1.7 


185 


1 


1.7 


Total Count 


58 


100% 



Fourteen-day trial periods were the most common mode and when combined with 15-day trial periods accounted for 
27.6% of the trial period policies. The next most frequent were 28- and 30-day trial periods, which accounted for 
another 27.6% of the sample. Thus, approximately two and four week trial periods accounted for over half of the 
trial periods. This finding was supported by a smaller study surveying 32 virtual schools which identified two and 
four week trial periods as the most commonly reported (Watson and Ryan 2007). While several researchers have 
identified the need for consensus on common metrics such as course completion rates for quality assurance and 
comparison purposes (Pape et al. 2006; Watson and Ryan 2006; Watson et al. 2008), no one has proposed what that 
consensus would look like. Findings from this study and Watson and Ryan’s (2007) indicate that if accrediting 
bodies, policymakers, or virtual school coalitions were to attempt to standardize trial period lengths to enable 
comparisons, two or four week policies would likely meet less resistance than other trial period lengths based on 
their prevalence. There were several anomalies in trial periods presented by the data. There were two extreme 
outliers of 180 days and 185 days. When the descriptive statistics were calculated excluding these outliers, the 
adjusted mean was significantly smaller, at 19.61 days. 

Two schools were excluded from the data set. West Virginia Virtual School had a unique trial period policy 
in that the course provider determined it. In other words, West Virginia Virtual School served as a broker for a 
variety of contracted course providers, each of which might have had different trial period policies. Grigg’s 
University and International Academy, a private virtual school, had a trial period policy where a student could 
withdraw at anytime so long as they had not started the course. These policies further illustrated the variety in trial 
period policies. 

It is interesting to note what drives administrators in determining their trial period policies. Though not the 
focus of our study, one school administrator revealed that her school’s trial period recently changed from two weeks 
to four weeks to match the trial period length of another major virtual school. This was done in an effort boost the 
school’s completion rates to make them appear on par with a rival institution. This logic may explain why schools 
cluster around trial period lengths of two and four weeks. Virtual schools may be adopting the trial period policy of 
their nearest competitor in an effort to attract or retain students. Literature in regional policy diffusion has identified 
three primary reasons for policy diffusion across states: competition, convenience, and policy communities (Cohen- 
Vogel, Ingle, Levine, and Spence 2008). Economic (Dye 1990) and normative (McLendon 2003) competition where 
one adopts a policy to gain advantage over another state, to avoid disadvantage of another state, or to conform/keep 
pace with other states provides a plausible explanation for interstate policy adoption. Another factor playing into 
policy adoption may be convenience. Neighboring states or institutions may adopt another’s policy due to 
incomplete information, time constraints, or its appearance as a sound model (Cohen- Vogel, et al. 2008). Lastly, 
policies can be adopted as a result of an influential, community network with common values (Mintrom and Vergari 
1998). Future research could investigate the driving forces behind trial period policy length and adoption. 

University, private, state-led, and consortium schools had on average longer trial periods than other types of 
virtual schools with mean lengths of 30.5, 21.6, 21.13, and 19.6 days respectively. In contrast, cyber charter and 
single-district schools had 16.78- and 15.5-day periods. Flowever, a one-way ANOVA indicated that there was no 
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statistically significant difference between trial period lengths and virtual school type. There were also differences in 
the trial period lengths for schools located in different geographical areas. Northeastern and Western region states 
tended to have slightly longer trial periods. However, a one-way ANOVA indicated that differences between trial 
length and geographical region were not statistically significant at p=. 05. 

In summary, there was a wide range of trial period policy practices among virtual schools. These 
differences were not dependent on the type of virtual school, nor its geographical location. These findings compound 
the difficulty policy makers, schools, prospective students, and parents face when comparing completion rates to and 
between virtual schools and brick-and-mortar schools. 

Question 2: Course Completion Definitions 

All but two of the 86 schools provided qualitative definitions of a completion. These ranged from 
“remaining in the course regardless of the grade” to passing the course with an A- or better. Responses fell into three 
main categories: (1) completion of the course within the allotted time frame regardless of the grade received, (2) 
completion of the course within an allotted time frame with a passing grade, and (3) deferring judgment to the brick 
and mortar school the student physically attends. Within these main categories there were several more refined 
definitions. Table 5 illustrates the range of definitions as well as the frequency counts of each. 

Table 5. Frequency count of course completion definitions 



Definitions 


n 


% of sample 


Final grade is irrelevant 






Remaining in the course for its duration regardless of the final grade received 


16 


19.0 


Completing of all or the majority of the coursework and/or exam within the 
allotted timeframe regardless of the final grade received 


11 


13.0 


Mastery of material (not defined as grade or percentage) 
Final grade is determining factor 


1 


1.2 


Passing the course 


37 


44.0 


Passing the course and passing the final 


2 


2.3 


Completing four quarters with 60% or better 


2 


2.3 


Passing the course with a D/64% or better 


1 


1.2 


Passing the course with a C-/70% or better 


6 


7.1 


Passing the course with a B-/80% or better 


4 


4.8 


Passing the course with an A-/90% or better 


1 


1.2 


Deferred judgment 






Brick and mortar school the student attends defines a course completion 


3 


3.6 


Total 


84 


100% 



Thirty-three percent defined a course completion as completing a course regardless of the grade received. In 
contrast, 62.9% used a more stringent definition of a passing grade to indicate a completion. This finding was 
consistent by Watson and Ryan’s 2007 survey of 32 schools. Similarly, they found that roughly 66% of the schools 
defined a course completion as receiving a passing grade. 

Some schools had unique factors contributing to how and whether to calculate course completions. While 
they had internal policies that govern completion calculations, many deferred judgment to the student’s brick-and- 
mortar school. Other schools did not calculate course completion rates, particularly if the program was only for fill- 
time students, as was the case with one full-time charter school surveyed. 

To examine the relationship between course completion definitions and school type and geographical 
location, we ran a cross tabulation and Chi Square Test for Independence with p=. 05. This test indicated no 
significant relationships between course completion policies and types of virtual schools or the geographical 
location of the schools. 

The wide variation in how schools defined a course completion has significant implications. First, as with 
trial period lengths, idiosyncratic policies make it difficult for lawmakers, researchers, parents, and students to make 
meaningful comparisons among virtual school attrition/retention rates (Watson and Ryan 2006). If all other variables 
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influencing student academic success were equal, schools with less stringent definitions of a course completion 
would have lower attrition rates than virtual schools with more stringent definitions of a completion. Thus, attrition 
rates could be manipulated by factors unrelated to student learning. Second, if course completion rates were an 
indicator of the success and health of a course, then counting failing students in this statistic was illogical and 
counterproductive. A third of schools in our study, and in Watson and Ryan’s (2007) smaller study, counted students 
as successfully completing a course regardless of the grade or effort students put forth. Consequently, a student 
could sign up, never officially withdraw, mentally check out after the first week, and ultimately contribute to a 
higher course completion rate. Such a practice pollutes the metric rendering it useless as an indicator of the quality 
of the course, teaching, and/or instruction. 

Conclusions and Implications 

In summary, our study examined the variation in trial period and course completion policies — two policy 
factors that affect course completion rates. We found that trial periods were a fairly common practice among virtual 
schools and varied in length. We also found that the majority of schools defined a successful course completion as 
passing the course, though there was wide variation in what percentage constituted a “passing grade.” Consequently, 
it was impossible to compare completion rates of one virtual school to another. In order to make course completion 
rates a meaningful measure a standardized method for calculating this metric needs to be established and adopted 
throughout the K-12 online learning community. 

There are several implications that arose from this study. The variation in trial period length and course 
completion definitions gave weight to earlier calls to standardize course completion metrics (Smith et al. 2005; Pape 
et al. 2006; Watson and Ryan 2007). Though schools can have their internal policies for calculating course 
completions, for statistical and reporting purposes, all virtual schools should be required to start counting students at 
the same time and in the same manner. Ideally, when and how students are counted ought to be similar to how brick 
and mortar schools calculate course completions, thus allowing parents, prospective students, researchers, and 
legislators the ability to compare virtual school attrition rates with other virtual school and with brick-and-mortar 
school completion rates. 

Since there was no statistically significant difference in trial period and course completion policies by 
geographical location and only state-wide supplemental virtual schools were unique in completion definitions, 
regional accrediting bodies ought to be able to agree on a single, nationwide trial period length and course 
completion definition to facilitate meaningful comparisons. To reinstitute course completion rates a quality indicator, 
virtual schools should abandon the practice of including failing students. Accrediting bodies or legislatures could 
mandate this change in practice. Unless virtual schools were required to this, there would be no incentives for 
schools to change. 

Future research should examine students who are withdrawing from virtual school programs during this 
trial period and how they compares to the types of students who remain in the course beyond the trial period. This 
would be a meaningful research question, as it speaks to the question of which students online learning is actually 
serving. Findings from such research could help guide practitioners in developing instructional supports to assist 
those students who struggle in these online learning environments. 
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Appendix A 

1 . Does your virtual school allow students to register for a class and drop that class without being counted in the 
official attrition statistics? 

2. If you answered yes to question 1, how long do students have in this "trial period"? 

3. How does you virtual school define a completion? 
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Abstract 

The study examined a professional development framework that resulted in improved student achievement and 
capacity building for systemic instructional improvement. Using a data-driven professional development plan, 
teachers’ received intensive training on a Course Management System (CMS), administrative follow-up, technical 
support, self-evaluated products, and extrinsic motivation to practice and continue to self-develop materials. This 
year long, extrinsically rewarded job-embedded aspect resulted in the continuity of implementation and resulting 
impact on student achievement. The progression from a half-loop professional development model to a double-loop 
cycle demonstrated an often seen process whereby a self-organizing system, guided by research can become more 
than the project directors intentionally planned or expected. 

Introduction 

This paper presents a model for systemic transformation in schools, based on the concept that 
current school systems do not dynamically adjust to the needs of our current information-rich society. This model is 
based partly on Alan Bain’s (2007) Self-Organizing Schools Model and research. Self-organizing dynamical 
systems are discussed as one solution, in particular Bain’s Model. A project is presented that demonstrated the use 
of a CMS Course Management System (CMS) to facilitate positive distributed change through the implementation 
of self-directed learning supplements and social networking approaches across partnering districts. This 
demonstrated one vehicle for assisting schools in developing self-organizing schools through professional 
development networks. 

Background 

School organizations primarily exist today as self-sustaining structures driven by pre-existing 
structures and processes that meet the political and legislative needs of the local (LEA) and state educational 
agencies (SEA). This model was developed at the turn of the 19th Century to meet the needs of a growing, global 
industrialized society (Watson & Reigeluth, 2008). Just as the nine-month schedule of American schools was set by 
the needs of primarily agrarian society, the economic needs of the early 1900’s required a uniform product: the 
factory worker, who could follow simple directions, achieved a minimal skill set in the three R’s, and who was 
essentially interchangeable with another worker. The current world in which today’s students find themselves also 
requires certain prerequisite skills. But it requires the ability to dynamically adapt to an ever-changing flow of new 
information that is developed, shared, and modified through social networking tools. Whether schools have begun to 
develop classes without walls, dynamic informational systems and knowledge management tools have started 
tearing those walls down in virtual space. Given the wide accessibility and usage of such technologies among 
school-age children, schools must transform from static systems to dynamic organisms than can adapt to a quickly 
transforming world; a world with growing opportunities for students to find, share, and distribute information with a 
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wide range of students, teachers, and content experts living in a dynamical, virtual knowledge space. In order to 
adapt to such a world, schools allow emergent qualities and capabilities to thrive and develop among students and 
teachers and administrators embrace such change. 

Bain (2007) reported such emergent qualities only thrive in a self-organizing system. From 1992-2003, 

Bain conducted a program to develop and sustain a self-organizing school reform project. The basic tenets were that 
such a sustainable system needed: simple rules, a design that is embedded in the process of the school, the use of 
emergent feedback that is bottom-up, has similarity of scale at all levels throughout the system by using various 
teams distributed throughout, and uses dispersed control rather than centralized. In Bain’s model, the technology for 
cooperative learning and knowledge management is adopted and scaled up through distributed teams that self- 
organized at different levels of scale. Emergent feedback eventually scales up to lead to district-level professional 
development changes allow enterprise level adoption. The self-organizing school project reported impressive results 
on SAT-I scores. The effect size for overall performance improvement was .58 and .70 for students with learning 
disabilities. This is considerably large than found with most comprehensive school reform programs. Part of the 
difference with such a model may be attributed a more transformational change than just tweaking the system. 

Bartunek and Moch (1987) referred to change that just tweaks the existing systems structure as 1st Order 
Change. Substantial reformation of the system is 2nd Order Change. An actual systemic transformation of the 
systems is a 3rd Order Change that must be sustainable and builds change management into the transformed system. 

Professional Development 

Professional development can have many direct and indirect effects on teachers. For example, Wenglinsky 
(2002) pointed out that professional development (PD) directly affected the classroom practices of the teachers. The 
biggest challenge Guskey (2003) and Killion (2002) reported that classroom teachers did not integrate new training 
into their instructional bag of tricks. This has been one of the noted weaknesses in dollars and time spent on in- 
service teacher professional development. 

Technology integration in PD has had limited research. Holum and Gahala (2001) suggested that 

1. Professional development must be on-going, 

2. Faculty must have access to technology tools, and 

3. Professional development should focus on how to use technology tools in ways that are relevant to the 

teacher. 

Teachers need to have frequent opportunities to practice using new technology at a level appropriate for them 
(Cook & Fine, 1997), and relevant to the content that they teach. A one-shot professional development workshop on 
technology innovations has limited chances of transferring to the classroom if there is limited or delayed opportunity 
to practice those skills in classroom. 

Often teachers learn new technology in isolation or with tutoring with a technology mentor. There tends 
not to be a true community of support for technology integration. What community exists usually does not extend 
past the school building or the district. Adoption of technology for teaching is also often top-down. As Buckingham 
(2007) stated, 

As we have seen, there are many ‘local’ reasons why technology has failed to transform learning in the 

manner that many of its advocates have envisaged. Y et perhaps the overriding one is to do with the 

centralized, top-down nature of the innovation itself (p. 64). 

This top-down enforcing of technology innovations maintained a static structure and environment that 
perpetuates its form and processes. What was viewed as needed was a more dynamical process that involved more 
stakeholders, created a more adaptive, organic system that processes feedback from many stakeholders at varying 
levels of organization structure. This leads to a more responsive system where participants reacted to their 
environment (stakeholders, external policies and regulations, economy, and a rapidly changing technology-infused 
landscape). Such two-way feedback also encouraged more of a sense of ownership within the participants. They 
became shareholders and change agents within this educational organism that could extend and interact with other 
educators outside the limits of the school or even the district. 

The professional development model used workshop and post-workshop activities that helped educators in 
rural, Midwestern secondary schools to design, create, and use online course materials and alternative online 
strategies for delivering content that encourages student collaboration and participation. The model followed the 
theoretical frameworks focused on the professional development research and Vygotsky’s (1978) zone of proximal 
development and a combination of parts of other research on professional development (Guskey, 2003; Fullan, 

1996; Galloway, n.d.; Killion, 2002). 

Traditional professional development models represented combinations of half-loop activities. These 
included such administrator-planned non-productive activities as having external expert train teachers in a single all 
day or half-day event formats. Administration checked event as being completed and the teachers trained. There was 



72 




minimal regard for follow-up or sustainability. Typically the building principal did not even participate in the 
training, therefore could not have followed-up, even if he or she had wanted to. A fundamental belief of the 
traditional model of professional development was that training was not really a means to an end, but a way to 
occupy state mandated teacher in-service trainings. There was also not an expectation for teachers, once trained, to 
use their training. The system did not encourage implementation. The illustration of a half-loop model was shown 
in Figure 1. 




Teachers do not 
use training in their 
class. 



Figure 1. Traditional professional development on technology. 

As the research on K-12 public school professional development represented new ideas on effective 
methodologies, some districts began to implement a single loop system (see Figure 2). In this top down model, 
administrators continued to determine the needs of the teachers in their buildings or district and trained teachers 
through the use of external experts. The training continued to be a sit and get experience but administrators 
completed differing degrees of follow-up. Administrators would complete observational feedback and provide 
instructional support. As a result of the principal’s follow-up activities, teachers used the strategy(s) or skill(s) 
acquired from the training. The implementation was not sustained over time but dissipated when then the follow-up 
activities were discontinued. Principals did not complete the loop because of a belief in their significance to the 
outcome but rather because it was expected of them. Walk-through observations and or other supportive activities 
were also influenced by the recency effect (Glanzer & Cunitz, 1966). That which was the most recent training 
experience would receive administrative attention and support. The farther from the date of the training event, the 
less time administrator’s spent on providing teachers’ follow-up support. Once short-lived attention had been given 
to follow-up and teachers were observed using their training, principals were ready for teachers to receive additional 
training. In many cases the next phase of training may not relate or build on to the previous training. Teachers were 
vague in their understanding of any relationship and the loop repeated itself. 



Administrators PLAN PD 



Teachers use skills 
or strategies while 
being observed 



Teacher's training is sit &. get 



Administrators conduct walk- 
throughs & other follow-up 
activities 



Figure 2. Considered progressive professional development model. 

The results of these models were wasted resources, no positive impact on student achievement; increased 
frustrations by teachers wanting to see better uses of their time and limited resources; and disconnects between 
administration, training, research, and the teachers. The study focused on creating a single loop that would improve 
student achievement and increase the likelihood that teachers would sustain their newly acquired skills. 
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Methods 

The research question asked the following questions: How can districts get teachers to utilize the 
knowledge and skills acquired from in-district professional development on a course management system? Teachers 
from seven different districts developed course supplements to be implemented through a course management 
system (CMS). Aspects of the model included: (a) Math and science teachers were paid small stipends to complete 3 
eight-hour training on the CMS. (b) Administrators were trained in observational and feedback strategies. They 
were also shown examples of high quality online courses, (c) Teachers were given a rubric for self-evaluation on the 
components of the course materials they created, (d) On-going technical support was offered between training 
sessions, (e) It was the realization that teacher’s needed incentives to continue their practice and implementation past 
the point of external accountability, that supported the pay for product incentive opportunities. To reinforce the 
development of independence with the CMS teachers were offered an additional small stipend for each instructional 
unit created. This gave teachers an incentive to practice until two things happened: they became somewhat proficient 
with their skills using their software and they 

The CMS training was part of a continuous improvement framework model. Teachers from different 
districts came to together to learn CMS and social networking tools to develop innovative activities to supplement 
their courses. The participants then shared these supplements with fellow teachers from other districts who provide 
feedback and gained insights on how they might apply the demonstrated strategies to their own school systems and 
content areas. They thus became agents of change for their own districts. 

As progressive districts began to calculate the real dollar expenditures related to teacher training, federal 
grant programs expected scientifically-evidenced based instruction, and academic accountability was becoming as 
visible as athletic accountability, districts began looking for more effective professional development models. 
Research supported the following process (see Figure 3): 



1. PD Plan 

Teachers, Data, Administrators 



, / 

6. Using data, evaluate 
the PD Plan, modify 
accordingly 



2. Teachers’ and 
administrators’ training is on- 
going, 



5. Extrinsic reward = 
sustainability 

\ ) 



3. Administrators follow-up 
and support 

k J 



4. Teacher self-evaluation 



Figure 3. Single-Loop Professional Development Framework. 

Step 1 . All teachers were invited to input ideas into the development of the professional development plan. 
They were asked what they wanted and what they needed to tie their instruction to the 21 st century. Teachers 
identified a need for extensive training on online course management systems. Although they were not clear about 
the exact what, they were certain that in order to become 21 st teachers their students needed to access structured 
lessons - using multi-media, self-assessments, application based studies, and much more. The teachers also insisted 
that their principals have some understanding of the technology hardware and software being used AND that they 
understood how to evaluate their instruction. These needs were then aligned to the district test data and other data 
sets to determine a professional development plan. Teachers determined the training schedule and the feedback 
loops. 

Step 2. It was important to the project that teachers did not miss instructional time with their students. The 
trainings were scheduled over time, after school, weekends, and summer. The training was conducted by an external 
expert in both the technology required and effective instructional design. The principals received training focused on 
effective math and science instruction, as well as training in the criteria for online course evaluation systems. 

Step 3. Teachers were trained in the CMS software. Training protocol used chunking. Small pieces of 
required knowledge and skills were shared, practiced, and implemented. As teacher confidence grew, smaller 
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chunks were grouped into larger projects. Teacher modeling lessons, observing lessons/evaluating materials, team- 
teaching/creating materials, 

Step 4. - Teachers and administrators were trained in the CMS online performance-based rubric. This 
instrument, adapted from the Illinois Online Network (2007) ’s Quality Online Course Initiative Rubric, gave a 
baseline for quality expectations. Teachers were grateful for concrete expectations and examples of excellence. As 
one teacher stated, 

We usually find out we have done something wrong afterwards - knowing expectations upfront 
and having something concrete by which to self-evaluate really helped me structure stronger 
supplemental materials. It was also important knowing that others would be evaluating my work. 

Step 5. Once teachers were trained they were offered an extrinsic incentive for continued practice. 

Teachers were paid $300 for each additional unit of supplemental materials created. These were not single 
lesson but comprehensively supporting an entire unit. This was done to support on-going practice and 
performance. Teachers shared their products with each other - creating a learning community. 

Step 6. Student data &/or performances, teacher self-reflections, and rubrics were used to determine 
effectiveness . 

Schools involved in the grants-sponsored program reported improvement in their district’s state 
achievement test scores. Positive gain scores were reported in math, science, and reading test scores for Latino and 
special education students at the participating schools. The utilization of technology to foster collaboration between 
the teachers and within their classes should be acknowledged as having a partial responsibility (Hemphill & McCaw, 
2009). An unexpected result of the new framework was the development of a double-loop system. Loop one 
included the project framework and loop two resulted from the teacher behaviors of the extrinsically motivated 
teachers (see Figure 4). 



L 



/ 



Loop 2 CMS embedded 
into teaching bag of 
tools 



7 



Loop 2 Peer 
Feedback 







Loop 2 Practice makes 
permanent 



Figure 4. Double-Loop Professional Development Model (McCaw Model). 

Over time, those teachers extrinsically motivated by small stipends were found to sustain over time the 
knowledge and skills used within the CMS until they became embedded within their bag of instructional tools. 
Teachers then used their own data to modify, improve, and update existing materials; while continuing to create new 
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ones. These new ones were not a result of extrinsic rewards but the intrinsic ones identified with excellence in 
teaching. 

Often in research outcomes can be surprising. This was found to be true in this project. As the system self- 
organized it evolved. All of the previous steps applied but outcomes of the extrinsic rewards became the fulcrum 
point for energizing newer deeper levels of self-analysis, communications with peers, greater integrity to standards 
of excellence, growing willingness to mentor each other and to listen to the voices of their students. Each of the 
subsequent parts has the potential for extending the life cycle of the model. In Loop 2 as the Feedback process grows 
it could potentially become its own loop - sustaining the process with energies experienced with experimentation 
and subsequent success. 

The $300 stipend for each completed unit seemed to give enough impetus to the system for it to become an 
extension Loop. Teachers felt obligated to complete what they had started - less for the money than for the desire to 
finish what they had started. The stipend tied their feelings of obligation to something more concrete than time. The 
money seemed to also play a role in justifying their sense of being appreciated and supported rather than simply 
expected to become better. All of the teachers admitted that the stipend did not cover the enormity of time that was 
involved in the creation of each unit. Another aspect that created energy was the concreteness by which their 
projects were to be evaluated. The rubric explicitly identified traits and expectations for excellence in creating and 
teaching an online course. The specificity gave them a clear path to follow. This path and the stipend created a 
venue within which they created quality course supplements. With each lesson and then unit teachers became more 
confident. With confidence came increased speed in design and implementation. As more of their courses used the 
supplemental materials student feedback increased. Teachers reported that students were commenting on the 
materials, surprised by the variety of tools, opportunities for practice assessments, and related materials to the 
subject area. One teacher shared that some of her students were impressed that there were teachers finally teaching 
for their generation - using technology not just lecture, blackboards, and paper pencil. The teacher literally shined as 
she shared her student’s perceptions of her. With practice and permanency came the point where teachers were 
automatically engaging with the CMS as a co-teacher. The practices, knowledge, and skills had become embedded 
into their bag of instructional tools. The teacher and the tools were felt to be one in the same. 

Findings and Conclusions 

Findings can be reported at three levels: student achievement, continuation of teachers using the CMS, and 
effectiveness of the model in reflecting the processes used. Student achievement was shown to have gradual but 
systemic improvements. For example, percentage of all junior status students at the participating high school that 
met the state benchmark on the publicly accessible American College Test (ACT) math data: 2005 as 18.7%, 2006 
as 20%, 2007 as 26% and 2008 as 29%. Although there were other interventions incorporated into the overall school 
improvement processes, the use of the CMS was also found to increase over time. Increases were also found at the 
middle schools, grades sixth through eighth, especially with minority students. 

This association between the use of technology and student achievement was supported by the findings 
from a meta-analysis review by Smith, Clark, and Blomeyer (2005) of research in technology integration. The study 
demonstrated an effective process for conducting and sustaining the professional development of teachers when 
creating online course supplements. The findings add to, and are supported by the body of research on professional 
development and technology implementation in the schools. In a time of limited resources there is an increased need 
for cost-effective, evidence-based, and implementable training modules for educators. The study presented in this 
paper is an early contributor for meeting such demands and also illustrates the potential using course management 
systems and eventually learning management systems (Reigeluth, et ah, 2008). 

An additional conclusion related to the theories behind self-organizing systems. It is viewed as impossible 
to predict the outcomes of healthy research-based systems. It is not the limitations of systems but the system’s 
limitations that minimize their impact. The study provided evidence of these assumptions. Once the organism was 
allowed to grow from a half-loop into a single loop and supported with extrinsic rewards for teachers and intrinsic 
rewards for administration, the single loop became more. Teachers focused on establishing their own standards, 
elevated their levels of sharing ideas and strategies, and listened to many voices for improvements. Including in 
some cases the voice of the student. 

While the modest project presented in this paper was not a full implementation of some form of a self- 
organizing school system, it does demonstrate the effectiveness and impact of early technology adapters as change 
agents in the use of new knowledge management tools and social networking tools to transform schools into a form 
that more closely represents the world students will succeed and flourish in tomorrow. It hopefully illustrates a way 
to introduce and distribute the tools and processes that will facilitate a level-three change that will transform schools 
from a static top-down structure to one that is self-organizing and dynamic. 
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Abstract 

This study examined the relationship between teaching presence as defined by the Community of Inquiry 
model and student satisfaction. The participant group of 22 graduate students represented a convenience sample 
from four fully online college courses. Much has been written about the positive impact of social presence on 
student interaction and satisfaction in online courses, but recent research is focusing on the importance of teaching 
presence and its role in online learning environments. The teaching presence construct represents instructional 
design and course organization, facilitated discourse, and direct instruction. A correlational analysis found a 
significant positive relationship between teaching presence and student satisfaction, while no significant relationship 
was found between previous online course experience and teaching presence. T-test provided evidence that age has a 
significant effect on student perception of teaching presence, 

Introduction 

In recent years, as researchers investigate student learning in online environments, the phenomenon of 
online presence has emerged. Online presence refers to one’s ability, whether student or instructor, to project their 
human nature into an otherwise computer mediated environment, void of the visual and verbal cues found in the 
traditional classroom. “In an online course, the simplest definition of presence refers to a student's sense of being in 
and belonging in a course and the ability to interact with other students and an instructor although physical contact is 
not available” (Picciano, 2002, p. 22). 

The concept of presence in online learning environments can be further examined when several elements 
essential to educational transactions in the online environment are included. These elements are 1) cognitive 
presence, 2) social presence, and 3) teaching presence (Garrison, Anderson, & Archer, 2000). Garrison et al. (2000) 
define cognitive presence as the extent to which participants are able to construct meaning through sustained 
communication. Categories for cognitive presence include triggering events, exploration, integration, and resolution. 
Shea, Pickett, and Pelz (2003), elaborate the term by adding that it is achieved in concert with satisfactory social 
presence and effective teaching presence. Examples of cognitive presence include student questions, expressions of 
confusion, problem solving, and evidence of students making connections and building new knowledge. 

Kehrwald (2008) defines social presence as “. . . the means by which online participants inhabit virtual 
spaces and indicate not only their presence in the online environment but also their availability and willingness to 
engage in the communicative exchanges which constitute learning activity in these environments” (p. 94). Garrison 
et al. (2000) describe social presence as the ability to project oneself as a real person (one’s full personality) both 
socially and emotionally in the online environment. The original categories of social presence, as defined by 
Garrison and Arbaugh (2007) are affective expression, open communication, and group cohesion. They also suggest 
that activities which cultivate social presence can enhance the learner's satisfaction with the Internet as an 
educational delivery medium. Social presence can be recognized by the use of emoticons to express oneself, idioms, 
more informal and familiar language, and evidence of students helping each other in a supportive manner. 

Teaching presence as defined by Anderson, Rourke, Garrison, and Archer (200 1 ) is “. . .the design, 
facilitation, and direction of cognitive and social processes for the purpose of realizing personally meaningful and 
educationally worthwhile learning outcomes” (p. 5). Based on this definition, the authors distill three components or 
elements of teaching presence: instructional design and organization, facilitating discourse, and direct instruction. 
Examples of teaching presence are the instructor clarifying areas of confusion, correcting misinformation appearing 
in discussion postings, moving the discussion along and keeping it on topic, and encouraging students to participate 
online. Teaching presence focuses on the tasks and processes of teaching online; it is not the affective projection of 
the instructor’s self, or personality into the virtual learning environment per se. This is referred to by some as teacher 
social presence, where the instructor works to develop and nurture a sense in the students that there are real people 
who care about them and their learning in the virtual environment (Wise, Chang, Duffy, & del Valle, 2004, p. 249). 

However, Garrison, Anderson, and Archer (2000) acknowledge that for additional progress to be made in 
understanding the three components of teaching presence, more precise and valid indicators associated with each of 
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them are necessary. Shea, Picket, and Pelz (2003), address this by providing the necessary depth to these elements 
with the addition of sub-categories or indicators as depicted in table 1 . 

Table 1 

Subcategories or indicators of the main elements of teaching presence 

INSTRUCTIONAL DESIGN FACILITATING DISCOURSE DIRECT INSTRUCTION 

AND ORGANIZATION 



Setting curriculum 
Designing methods 

Establishing time parameters 

Utilizing the medium effectively 



Identifying areas of agreement and 
disagreement 

Seeking to reach consensus and 
understanding 

Encouraging, acknowledging, and 
reinforcing students’ contributions 

Setting the climate for learning 



Assessing the efficacy of the process 



Presenting content and question 
Focusing the discussion 

Summarizing the discussion 

Confirming understanding 
Diagnosing misperceptions 

Injecting knowledge from diverse 
sources 



Establishing netiquette Drawing in participants and 

prompting discussion 



Responding to technical concerns 

Using the indicators and their subcategories from table 1, Shea, Swan, Li and Pickett (2003) developed a 
Teaching Presence Scale (TPS) to tease apart the aspects of teaching presence from social presence and cognitive 
presence. A study using this scale was conducted to further examine the relationship between students’ perception of 
teaching presence and their satisfaction in fully online learning environments. The purpose of this research was to 
examine the indicators (elements) of teaching presence as they relate to student satisfaction in the hope of increasing 
instructor awareness about the importance of teaching presence and to guide faculty development initiatives. A 
closer examination of the three elements of teaching presence can help faculty design, facilitate, and instinct more 
effectively in the online environment. 

Instructional Design and Course Organization 

Before students ever enter an online class, faculty and instructors must prepare the structure or bones of the 
course in advance. Anderson et. al. (2001) describe the course design and organization element of teaching presence 
as the planning and creation of the structure, process, interaction, and evaluation aspects of the online class. Shea, 
Swan, Li and Pickett (2005) offer that setting the curriculum, utilizing the medium effectively, and establishing 
netiquette are also parts of effective online course design and organization. Whether using a learning management 
tool such as BlackBoard or WebCT or an open source system such as Moodle, thoughtfully planning the course 
design and organization are critical aspects of providing a satisfactory online learning experience. 

According to Anderson (2004), the design and organization of a course offers instructors the first 
opportunity to develop their teaching presence. The design of an online course provides the framework from which 
students access all content, including the syllabus, discussion questions, commentary, mini lectures, the gradebook, 
images, audio and video files, and dialogs with other students and the instructor. Having a sense of the courses’ 
grand design, learning objectives, and the related activities, reassures students that they can participate effectively 
online and achieve the learning objectives. Ensuring that students can quickly find the critical information they need 
to get started in the class reinforces their confidence that they can be successful in the online environment. 

Therefore, how this content is structured and made accessible informs a student’s first impression of the online 
course as well as first impressions of the instructor. 

In this respect, keeping content consistent, accessible and clearly labeled becomes a key component of 
teaching presence. Swan (2001) found in her research that the “greater the consistency among course modules, the 
more satisfied students were, the more they thought they learned, and the more interaction they thought they had 
with their instructors” (p. 318). Research by Henry and Meadows (2008) also supports this idea. 
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In an online course, students need to be able to find everything they need to be successful learners and how 
to do so easily. Even in well-organized courses it is not uncommon to find out, part way through the course, 
that one or more students have not found some of the essential information (Henry & Meadows, 2008, p. 
10 ). 

Providing timelines, organizational guidelines, clear expectations for student work and activities also 
support sound instructional design and course organization elements. Instilling teaching presence is also 
accomplished by explaining or modeling for students what appropriate responses look like from an etiquette or 
online professionalism perspective. 

Facilitated Discourse 

The second element of teaching presence, facilitated discourse, “. . . is the means by which learners develop 
their own thought processes, through the necessity of articulating them to others” (Anderson, 2004, p. 280). 
According to Shea, Swan, Li and Pickett (2005) the instructor task of facilitating discourse is necessary to maintain 
learner engagement and refers to a focused and sustained deliberation that differentiates it from a discussion. The 
authors characterize facilitated discourse by including such instructor tasks as identifying areas of agreement and 
disagreement in discussion postings, helping students understand and reach consensus, reinforcing and encouraging 
student contributions, setting the climate for learning, drawing in participants and assessing the efficacy of the 
process. The authors also found that a strong and active presence of the instructor, guiding the discourse, is related 
to both students’ sense of connectedness and learning. Facilitated discourse goes beyond the social aspects of 
making contact online. It offers a deeper, richer, student to instructor interaction relative to the content at hand, and 
it is distinct from the instructor participating in a chat room or informal blog with students. 

Appropriate facilitation is demonstrated when the instructor makes an overt effort to build trust and 
transparency, engage all students in online discussions, and model exemplar discussion postings and participation. 
Building trust online is key to creating a safe and open learning environment, one in which learners are free to 
explore, share and create knowledge. Trust allows for risk-taking, a necessary component in learning, while 
promoting sharing and mutual learning. Within the element of teaching presence, the instructor can also model 
expected and appropriate online discussion participation by posting her own thoughts or ideas about a particular 
topic and conclude by adding additional open ended questions to encourage further discourse. Trust is further 
demonstrated when students are asked to lead or moderate and facilitate the discussions postings. 

Other aspects of effective discourse facilitation include providing prompt feedback to student questions and 
postings, and using student names in communications. Posting positive remarks when students demonstrate 
understanding or reach consensus are also effective discourse facilitation strategies. Pointing out areas of 
disagreement or misconceptions may pique student interest and create cognitive dissonance, thereby stimulating 
others to participate. Being involved in student discussions and communications on a regular basis supports teaching 
presence. As Swan (2001) found in her empirical study, student perceptions of interaction with their instructors were 
highly correlated with both satisfaction and perceived learning. 

Direct Instruction 

Instructors and faculty demonstrate direct instruction when they provide intellectual and scholarly direction 
for the course, sharing their knowledge as content experts with students. As Anderson et. al. explain (2001), the 
teacher as subject matter expert “is expected to provide direct instruction by interjecting comments, referring 
students to information resources, and organizing activities that allow the students to construct the content in their 
own minds and personal contexts” (p. 9). Direct instruction communicates and sets the intellectual climate for the 
course, modeling the qualities of a scholar, including integrity and sensitivity (Anderson, 2004). In content rich units 
the instructor can present herself by including her personal reflections on content issues, providing relevant 
anecdotes and examples, and offering insights into being successful in an online learning environment. 

Shea et. al. (2003) found that instructor behaviors for facilitating discourse and direct instruction correlated 
highly with satisfaction and learning, supporting the notion that the instructor is expected to be the subject matter 
expert and is expected to share that knowledge in the learning environment in the form of direct instruction. Swan, 
Richardson, Ice, Garrison, Cleveland-Innes, and Arbaugh (2008) state, “Instructor responsibilities are to facilitate 
reflection and discourse by presenting content, using various means of assessment, and feedback” (p. 3). By 
providing thoughtful, prompt, direct instruction, faculty can enhance their teaching presence in the online 
environment. 

This study addressed the following research questions: 

1. What is the relationship between teaching presence as defined by the elements of instructional design 
and organization, discourse facilitation, and direct instruction, and student satisfaction in a fully online 
college course? 

2. What is the effect of age on student perception of teaching presence? 
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3. What is the relationship of the number of online classes a student has taken and their perception of 

teaching presence? 

Methodology 

Participants 

Participants represented a convenience sample of under graduate and graduate college students enrolled in 
four online classes during the first six-week session of summer semester 2009. All four classes were taught by full 
time PhD faculty. Twenty one students from a total of 71 students (30%) elected to participant in the online survey. 
These fully online classes are part of the teacher preparation and educational technology curriculum at a university 
located in the southwestern region of the United States. 

The survey used in this research was adapted from the validated Teaching Presence Scale (TPS) from Shea, 
Swan, Li, and Pickett (2005), with slight modifications; questions addressing the other two components of the 
Community of Inquiry construct (social presence and cognitive presence) were omitted and four open ended 
questions regarding student perceptions of teaching presence were added. 

Materials 

The survey instalment was an electronic questionnaire that consisted of 17 Likert measured items which 
corresponded to the three elements of teaching presence as defined by the Garrison et al. (2000) model. The survey 
opened with two questions directed toward learning more about the participant group: their age range and how many 
previous online courses they had taken. The 1 7 items were evaluated using a 5 point scale with 1 for strongly 
disagree through 5 for strongly agree. Included were two additional Likert scale questions designed to measure 
overall student satisfaction with the course using the same 5 point scale. The survey concluded with four open ended 
questions designed to elicit student opinions about course organization and instructor involvement 
Procedure 

During the fifth week of the six-week summer session, a hyperlink to the online survey along with a letter 
of consent was posted in each of the four online classes being studied. Students agreed to participate by clicking the 
survey link which took them to the informed consent letter and then into the actual survey. Students had one week to 
complete the survey during which two reminder emails were sent. All emails were sent through the university’s 
course management system. Participation was voluntary and survey data was collected anonymously by the survey 
provider. The survey was closed at the end of the sixth week. 

Data Analysis 

Two correlation computations using Pearson’s r were performed to answer the research questions 
addressing student satisfaction and previous online course experience with teaching presence. Correlation designs 
are used to determine the extent to which two or more variables are related among a single group of people, in this 
case the participant group of 22 students taking online courses. In correlational research there is no attempt to 
manipulate the variables or to prove cause and effect. A t-test was performed in order to examine the effect of age 
on students’ perception of teaching presence. All statistical calculations were completed using SPSS statistical 
software. 

Results 

The Pearson product moment correlation for the mean teaching presence score and the mean satisfaction 
score was statistically significant at r (18) = .719, p < .001, N= 19. The evidence suggests a significant positive 
relationship between students’ perception of teaching presence and their satisfaction with the class. 

Another Pearson product moment correlation was performed to answer research question three regarding 
the relationship of previous online courses taken and student perception of teaching presence. Evidence from these 
calculations suggests a small negative relationship between the number of online classes taken previously and the 
mean teaching presence score; r (18) = -.382 ,p = .107, N= 22. There is not a significant relationship between the 
number of online classes previously taken and the mean teaching presence score. 

A one sample t-test was performed to determine if age has an effect on student perception of teaching 
presence. This resulted in t{2\) = 9.253, p < .0001, N= 22, thereby suggesting that age had a significant effect on 
the mean teaching presence score (M = 4.1053, SD = .49006). 

Teaching Presence Open Ended Questions 

When asked what they liked most about the online course organization, five student comments expressed 
appreciation for the clear directions regarding how to complete their assignments as well as explicit due date 
information. Four student comments expressed appreciation for the examples the instructor provided which helped 
them better complete their assignments. One participant wrote, “1 liked that the assignments came with directions as 
well as examples. I also liked that for the most part the directions were very clear. The teacher was also prompt 
when posting assignments and available to email and responded in a timely and friendly manor [sic].” 
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When asked what they liked least about how the course was organized, students reported difficulties with 
the group work, such as communicating with their group and not being able to view the completed work of other 
groups (eight comments). Other students commented that they wanted more interaction from the online instructor 
(three comments). For example, “The instructor didn't really do any teaching. All he/she did was provide feedback 
on the materials we created.” 

Students responding to the question about what the instructor did to support their learning, said 
overwhelmingly they appreciated the prompt feedback and quick email responses (14 comments). Secondly, they 
said the clear examples provided helped them to learn (9 comments). One participant commented, “Fle/she provided 
great examples of [the] expectations for the course, and most of the time the directions were clear and precise.” 

The final question asked what the instructor could have done to support their learning. A few students said 
they would like the instructor to be more engaged with the class and “to teach more” (2 comments) while others said 
they wanted to see the work produced by other groups (2 comments). 

Discussion 

Because there is a strong positive relationship between teaching presence and student satisfaction, one can 
make a case for building and facilitating fully online courses based on the criteria of effective instructional design 
and course organization, facilitated discussion, and direct instruction. The results of this study support previous 
research (Swan & Shih, 2005; Shea, Pickett, & Pelz, 2003; Swan, 2001) demonstrating that student-teacher 
interaction and teaching presence were strongly related to student satisfaction and perceived learning. 

The instructional design and course organization element of teaching presence represents the structure, 
processes, interactions, and evaluation of the online course, tending to students’ needs regarding instructional design 
and organization means providing clear navigation paths within the course, providing clear instructions about how to 
participate in the online learning activities, as well as effectively communicating timelines, due dates, and 
expectations around polite and professional course participation (group norms or netiquette). 

Facilitating discourse requires the instructor to participate actively in student dialogs and discussion 
postings by raising questions, making observations and contributing ‘expert’ commentary that move the online 
conversation along. Previous research (Shea, Swan, Li, & Pickett, 2005; Shea, Pickett, & Pelz, 2003) found that 
evidence of a strong and active presence from the instructor, one where she actively guides the discourse, was 
related to students’ sense of learning. 

Direct instruction which involves content presentation, such as PowerPoint slides, mini lectures in the form 
of short audio fries, or text-based commentary highlighting salient points, as well as providing other sources of 
information (e.g. links to other online resources) also represents an important component of teaching presence. 
Explanatory feedback, delivered in a timely and constructive manner, along with the correction of 
misunderstandings, are additional ways faculty can enhance and support student satisfaction in the online learning 
environment. These combined elements when evidenced in an online class, underpin a successful and satisfactory 
learning experience for students as the positive correlation between teaching presence and student satisfaction found 
in this study suggests. 

As indicated by the results regarding the effect of age on the perception of teaching presence, there is a 
significant effect. In this study, the majority of students were between the ages of 17 -28 (54.54%), with only two in 
the class over the age of 40. These results may indicate that the younger online students have a greater need to 
experience teaching presence in order for them to stay engaged, motivated and satisfied in the online environment. 
Also the direction, feedback, facilitated discourse, and clear course design provide by teaching presence may be 
more welcomed by younger, less experienced online learners. 

When investigating the relationship of teaching presence and previous online course experience, the results 
suggests that there is no effect. This may be because students who have taken online classes previously are already 
familiar with the general processes, procedures, and expectations found in online course work and therefore do not 
have a need for teaching presence. It is also possible that because of their previous online experiences, these students 
are accustomed to teaching presence elements in online classes and take those elements for granted, or they have 
become proficient at being an online learner even if teaching presence elements are lacking. 

As Shea, Swan, Li, and Pickett (2005) found, students reported higher levels of teaching presence, learning, 
and satisfaction in courses where the instructors had been previously trained in how to establish effective teaching 
presence. The results of this study support a call for instructor and faculty professional development initiatives 
around building online learning environments grounded in effective instructional design and clear course 
organization, facilitated and meaningful discourse, and concise direct instruction; elements which enhance student 
engagement and satisfaction. 
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Limitations 

Unfortunately, the number of survey participants was small compared to the number of students invited to 
participate in the survey. This limits the generalizability of the results to a larger population. Additionally, this study 
did not review the specific course organization and design, facilitated discourse, and direct instruction found in each 
of the classes surveyed. More specific details, examples, and evidence of teaching presence within the online 
courses surveyed may better inform the results and implications for improving teaching presence. 

Areas for Future Research 

Research questions generally beget more research questions and this study is no exception. This study did 
not address the effect of teaching presence on learning, and it did not examine the relationship between teacher 
social presence and teaching presence although the two are closely related. Other areas for investigation include 
observing teachers who have high teaching presence scores and capturing how they facilitate online, the types of 
activities they offer students, and how they plan and structure their online courses. 

More recent studies (Arbaugh, 2007; Shea & Bidjerano, 2008) have examined teaching presence as just two 
factors or elements: directed facilitation and instructional design and organization. Shea and Bidjerano (2008) 
recommend making a clearer distinction among the factors defining direct instruction by adding such characteristics 
as the instructor’s ability to provide valuable analogies, useful illustrations, and present helpful examples (p. 552). 
Therefore, future studies on teaching presence may need to include a more comprehensive and revised description of 
the direct instruction element. 
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Survey Questions: 



Appendix A: Survey Instrument 



1. Choose your age range. 

17-22 

23-28 
29-34 
Over 35 

2. Choose which best represents your previous online course experience. 
This is my first online course. 

I have taken 1 online course. 

I have taken 2 or more online courses. 



Instructional Design and Organization 


Strongly 

Disagree 


Disagree 


Neutral 


Agree 


Strongly 

Agree 


1. The instructor clearly communicated important 
course topics. 


1 


2 


3 


4 


5 


2. The instructor for this course clearly communicated 
important course goals. 


1 


2 


3 


4 


5 


3. The instructor provided clear instructions on how to 
participate in course learning activities. 


1 


2 


3 


4 


5 


4. The instructor clearly communicated important due 
dates and time frames for learning activities. 


1 


2 


3 


4 


5 


Facilitating Discourse 


Strongly 

Disagree 


Disagree 


Neutral 


Agree 


Strongly 

Agree 


5. The instructor for this course was helpful in 
identifying areas of agreement and disagreement on 
course topics that helped me to learn. 


1 


2 


3 


4 


5 


6. The instructor was helpful in guiding the class 
towards understanding course topics in a way that 
helped me clarify my thinking. 


1 


2 


3 


4 


5 


7. The instructor helped to keep course participants 
engaged and participating in productive dialog. 


1 


2 


3 


4 


5 


8. The instructor helped keep course participants on 
task in a way that helped me to learn. 


1 


2 


3 


4 


5 


9. The instructor encouraged course participants to 
explore new concepts in this course. 


1 


2 


3 


4 


5 


10. Instructor actions reinforced the development of a 
sense of community among course participants. 


1 


2 


3 


4 


5 


Direct Instruction 


Strongly 

Disagree 


Disagree 


Neutral 


Agree 


Strongly 

Agree 


1 1 . The instructor provided useful illustrations that 
helped make the course content more understandable to 
me. . 


1 


2 


3 


4 


5 


12. The instructor presented helpful examples that 
allowed me to better understand the content of the 
course. 


1 


2 


3 


4 


5 


13. The instructor provided explanations or 
demonstrations to help me better understand the 
content of the course. 


1 


2 


3 


4 


5 


14. The instructor provided feedback to the class 


1 


2 


3 


4 


5 
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during the discussions or other activities that helped us 
learn. 












15. The instructor asked for feedback on how this 
course could be improved. 


1 


2 


3 


4 


5 


Overall Satisfaction 


Strongly 

Disagree 


Disagree 


Neutral 


Agree 


Strongly 

Agree 


16. Overall, I am very satisfied with this course. 


1 


2 


3 


4 


5 


1 7. Overall, this course is meeting my learning needs. 


1 


2 


3 


4 


5 



Open ended questions 

1. What did you like most about the way this course was setup and organized in Blackboard? 

2. What did you like least about the way this course was setup and organized in Blackboard? 

3. What are some things this instructor did to support your learning in the online environment? 

4. What are some thing this instructor could have done to support your learning in the online environment? 
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Abstract 

One essential question concerning the assessment of knowledge and mental models is that as to which methodology 
should be used. The main objective of our study is to track learners’ progression while using graphical and 
language-based externalization techniques. Specifically, we compare the similarities and differences of the 
graphically and textually elicited knowledge structures and their semantics. Our results reveal that graphical notes 
and texts re-represent different things. This supports our newly introduced model of information density vs. 
emphasis - from the different types of notation, we expect different types of information structures. 

Introduction 

Over the past decades, various researchers have contributed to the further development of the theoretical 
foundation of mental models (Al-Diban, 2002; Dinter, 1993; Gentner & Stevens, 1983; Johnson-Laird, 1989; 
Norman, 1983; Spector, Dennen, & Koszalka, 2006) and also to their application in instruction (Anzai & 

Yokoyama, 1984; Ifenthaler, Pirnay-Dummer, & Spector, 2008; Mayer, 1989; Seel, 1995, 2003). However, there are 
still a number of important unanswered questions concerning the assessment of mental models (Ifenthaler, 2008a; 
Ifenthaler & Seel, 2005; Seel, 1999). 

One essential question concerning the assessment of knowledge and mental models is that as to which 
methodology should be used. Many authors consider concept maps to be an adequate format of externalization for 
analyzing complex knowledge structures (Jonassen & Cho, 2008; Novak, 1998). Concept maps seem preferable to 
classical knowledge tests such as multiple choice tests for the purpose of representing linked knowledge by means of 
network-like visualization. On the other hand, there are strong arguments indicating that natural language 
representations are a good basis for assessing knowledge and mental models (Ifenthaler, 2008a; Pirnay-Dummer & 
Spector, 2008). 

The main objective of our study is to track learners’ progression while using graphical and language-based 
externalization techniques and compare the strengths and weaknesses of these assessment methods. Specifically, we 
will compare the similarities and differences of the graphically and textually elicited knowledge structures and their 
semantics. First, we will focus on learning and the role of internalization and externalization of knowledge 
structures. In the next section, we will introduce the design of our research study, followed by a brief overview of 
the applied graphical and language-based methodologies for the assessment and analysis of knowledge 
representations. Then we will present our results and discuss our findings. Finally, we will suggest a guideline for 
selecting a suitable methodology for different research demands and practical applications. 

Learning and the Role of Representations 

The theory of mental models has been the foundation for both facilitating and assessing learning (e.g. 
Ifenthaler, 2008a; Seel, 2003). Both areas of application are very close on a theoretical level because a mental model 
is a symbolic information processing construct. They are very closely related on the practical level, because in 
model-based learning and instruction, learning and assessment lead to the same design (Ifenthaler, Masduki, & Seel, 
2009). However, on the methodological level, assessment and learning differ due to differences between the 
internalization and externalization functions (Ifenthaler, in press). 

Internalization 

When we think of learning goals we think of internalization of features or properties of the learning 
environment. In many cases a learning environment is focused on a text, and the learner has a (limited) amount of 
time to spend with the text. In an ideal case, the learning environment is such that it facilitates the learner’s 
reconstruction (representation) of the environment, which exemplifies his or her real environment either 
symbolically (with text), through icons (pictures), or enactively - of course all of them with nuances. 
Externalization 

In assessment the mapping procedure is reversed. Learners have different options for externalization at 
their disposal using language (spoken and written) and graphical notes (abbreviations) of various commonly known 
(mind mapping, concept mapping) or idiosyncratic formats. Of course formal representations are also possible, but 
they are very rarely used in common learning tasks. Sometimes we ignore the fact that externalization itself 
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influences the learning outcome, not to mention the fact that the high demand for assessment also contributes to 
learning. Several new assessment methodologies have been developed in order to gain knowledge about what the 
learner does know rather than finding out what he or she doesn’t know - as is often the case in classical testing. The 
most important and most widespread sources for the assessment of what the learner knows are (1) text which the 
learner produces and (2) the learner’s graphical notes. 

In this study we look at two specific sources of externalized knowledge: text and graphical notes. Text that 
learners produce in assignments, such as work in class or homework, is an important source of their thought. The 
text can come from any task, from statements, comments, and online discussions to learning diaries and essays. In 
addition, learners can be asked to share their knowledge by using concept mapping techniques, in which they 
describe their ideas about a topic in graphical form (see Ifenthaler, 2008b; Novak, 1998). 

Different Approaches for Aggregation 

Both kinds of representation are used in everyday situations. Which is chosen depends on the purpose of 
the externalization. If the learner is just taking notes for himself, he may leave out significant parts of the knowledge 
and use the notes as a memory aid in order to retrieve knowledge more quickly (e.g. Ifenthaler, 2008a). Due to new 
developments, both modes of externalization can be quantitatively compared to the externalized thought of experts 
(Ifenthaler, 2008b; Johnson, Ifenthaler, Pirnay-Dummer, & Spector, 2009; Pirnay-Dummer, 2007). 

In order to be compared, knowledge has to be aggregated (condensed) to the most important and most 
relevant re -representation model. Knowledge is usually widely connected and associated. When the learner takes 
graphical notes as part of an assessment, he or she tries to aggregate the knowledge by him or her before the analysis 
- but almost nothing is known about the inter-rater aggregation stability within this process. When computer- 
linguistic methodology is used to aggregate the re-representation models from written or even spoken text, this will 
by definition raise the objectivity of the aggregation. The aggregation functions are always known down to the 
algorithmic level before they can be implemented into computer programs. However, the fact that learners implicitly 
tag the information which is most important to them may result in a loss of objectivity. 

Whereas classical methodologies are used to let the learner (or expert) aggregate his or her knowledge 
graphically for analysis, natural language-oriented methodologies like T-MITOCAR (Text Model Inspection Trace 
of Concepts and Relations) use the text itself to generate the graph (Pirnay-Dummer, Ifenthaler, & Johnson, 2008). 
T-MITOCAR can elicit, visualize, analyze, and compare knowledge structures of groups and individuals within a 
specific subject domain on the basis of natural language. It can easily interface with other automated analysis tools, 
e.g. with the SMD Technology (Ifenthaler, 2008b), or HIMATT (Highly Integrated Model Assessment Technology 
and Tools; Pirnay-Dummer, Ifenthaler, & Spector, 2009). 

Different Approaches for Aggregation 

However, it makes a great difference for the type and format of the representation whether the learner 
constructs the notes only for himself (e.g. for later remembering) or for somebody else’s understanding. If he 
constructs an externalization for himself, he may leave important things out which are obvious to him or use 
notations which are idiosyncratic to his understanding. If the goal is to inform somebody else using the notes, more 
care has to be put into notation standards and also more information generally has to be provided. Thus confronted 
with assessment situations, the learner will need to know that somebody else will have to make sense out of his 
notes. But this again shifts the intentionally inconspicuous assessment, in which the learner is not distracted by the 
assessment, to a more obvious situation in which the learner knows that he is being assessed. In most practical cases, 
the assessment may therefore be hidden in a learning task where, e.g., a case is presented in such way that it requires 
the information of other (existing or virtual) persons on the subject domain. 

Different Approaches for Aggregation 

From the different types of notation, we expect different types of information structures. When the learner 
conducts the aggregation (into the graph), he will have much more control over what is to be specifically 
emphasized. But if the learner writes the externalization into a text, the output will obviously have a higher 
information density and the emphasis will be more syntactically hidden in the text. On the other hand, a stronger 
emphasis on the individual learners’ aggregation will lead to more diversity in the actual re -representations, which 
will lower both the objectivity and the reliability of the assessment (see Figure 1). 



87 





Figure 1: Assumptions on density and emphasis for different aggregation methods 

Research Questions and Hypotheses 

To gain more information about the differences between self-aggregation and linguistic aggregation we 
conducted a cross-methodology study. The research goal was to show the differences and similarities between these 
very different assessment techniques. This will make it possible to provide a guideline for determining which 
methodology is best for various research questions and practical applications. Differences in internal coherence and 
cross validity are therefore of key interest in this investigation. 

The following set of hypotheses was tested in our experimental study. In order to understand the strengths 
and weaknesses of the methodologies, we first need to know how good the internal consistencies within each 
methodology are and how much they differ. Based on the different kinds of aggregations we expect the internal 
consistency to be different between the methodologies: 

H u : The graphical model representations and the textual model representations have different errors of 

variance. 

Hi 0 : The graphical model representations and the textual model representations have the same errors of 

variance. 

As described above, we took both texts and concept maps from the subjects. We took the concept map after 
the subject had written the text. The text was aggregated using HIMATT (Pirnay-Dummer et al., 2009). Due to the 
close assessment and the short time between writing and concept mapping we expected a close match between the 
structural and semantic indices for both methods (a description of all of the applied indices will be provided in the 
following section): 

H 2 .i: The indices of the graphical model representations and the indices from the text-based model 
representations correlate significantly with r > .7. 

H 2 0 : The indices of the graphical model representations and the indices from the text-based model 
representations correlate with r < .7. 

Method 

Participants 

Forty-seven students (31 female and 16 male) from a European University participated in the study. Their 
average age was 25.7 years ( SD = 1.7). All students were attending an introductory course on research methods. 

Materials 

Data were collected through graphical and text-based representations. For the graphical representations we 
used the software CmapTooIs (Canas et al., 2004). According to Novak (1998), a concept map is a graphical two- 
dimensional representation of communicated knowledge and its underlying structure. A concept map consists of 
concepts (graph theory: vertices) and relations (graph theory: edges). Research studies on the application of 
CmapTooIs indicate a wide acceptance of the software (e.g. Coffey et al., 2003; Derbentseva, Safayeni, & Canas, 
2004). The text-based representations are tracked with HIMATT (Highly Integrated Model Assessment Technology 
and Tools; Pirnay-Dummer et al., 2009). HIMATT tracks the association of concepts from a text directly to a graph, 
using a heuristic to do so. Texts contain model structures. Closer relations tend to be presented closer within a text. 
This does not necessarily work within single sentences, since syntax is more expressive and complex. But with texts 
which contain 350 or more words, associative networks can be generated as graphs from text and the heuristic 
becomes highly stable. The re-representation process is carried out automatically in multiple computer-linguistic 
stages (Pirnay-Dummer & lfenthaler, in press). 





Procedure 

The study was realized as a design experiment. It took place in a research methods course and consisted of 
(1) a 60 minute introductory phase, (2) a phase of learning with five learning tasks, and (3) a phase for assessing the 
final task performance. In the first phase, the participants were introduced to using the CmapTools software (see 
Canas et al., 2004) and writing text-based explanations of the phenomenon in question. Additionally, the instructor 
collected demographic data and delivered documentation on concept maps and text production, including examples. 
The second phase lasted several weeks. In this phase, the participants were asked to create an open concept map and 
a text-based explanation relating to their understanding of research skills. In total, five identical task explanations 
had to be uploaded at a specified date and time during the course. In the last phase, the final task performance was 
measured through (1) five written assignments, (2) a written exam, and (3) a written research proposal. The score of 
the final learning outcome was rated between 0 and 100 points. 

After uploading the concept maps and text-based representations, the instructor gave the students brief 
feedback to notify them that their assignments had been successfully uploaded and that they should carry on with 
their studies in the course. As we used open concept maps in our research study, the subjects were not limited to 
specific words while annotating the concepts and relations. 

Analysis 

To analyze the participants’ concept maps and texts, we used the seven core measures implemented in 
HIMATT (Pirnay-Dummer & Ifenthaler, in press; Pirnay-Dummer et al., 2009). Figure 2 shows the seven similarity 
measures of HIMATT, which include four structural indicators and three semantic indicators. The similarity 
indicators range from 0 (no similarity) to 1 (complete similarity) for better in-between comparability. Matrices of 
multiple representations can be compared simultaneously. Reliability indices exist for the single instruments 
integrated into HIMATT. They range from r=.79 to r=.94 and are tested for the semantic and structural measures 
separately and across different knowledge domains (see Pirnay-Dummer et al., 2009). Validity measures are also 
reported separately for the structural and semantic indices. Convergent validity lies between r=,71 and r=.91 for 
semantic comparison indices and between r=.48 and .79 for structural comparison indices (see Pirnay-Dummer et 
al., 2009). 
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Figure 2: HIMATT structural and semantic measures 



These seven measures are defined as follows (see Ifenthaler, 2008b; Pirnay-Dummer, 2007; Pirnay- 
Dummer & Ifenthaler, in press; Pirnay-Dummer et al., 2009): 

Surface Matching (R): The surface measure (Ifenthaler, 2008b) compares the number of vertices within two 
graphs. It is a simple and easy way to calculate values for surface complexity. 

Graphical Matching (G): The graphical matching (Ifenthaler, 2008b) compares the diameters of the 
spanning trees of the graphs, which is an indicator for the range of conceptual knowledge. It corresponds to 
structural matching as it is also a measure for structural complexity only. 

Gamma Matching (D): The gamma or density of vertices (Pirnay-Dummer, 2007) describes the quotient of 
terms per vertex within a graph. Since both graphs which connect every term with each other term (everything with 
everything) and graphs which only connect pairs of terms can be considered weak models, a medium density is 
expected for most good working models. 
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Structural Matching (S): The structural matching (Pirnay-Dummer, 2007) compares the complete 
structures of two graphs without regard to their content. This measure is necessary for all hypotheses which make 
assumptions about general features of structure (e.g. assumptions which state that expert knowledge is structured 
differently from novice knowledge). 

Concept Matching (C): Concept matching (Pirnay-Dummer, 2007) compares the sets of concepts (vertices) 
within a graph to determine the use of terms. This measure is especially important for different groups which operate 
in the same domain (e.g. using the same textbook). It determines differences in language use between the models. 

Propositional Matching (P): The propositional matching (Ifenthaler, 2008b) value compares only fully 
identical propositions between two graphs. It is a good measure for quantifying semantic similarity between two 
graphs. 

Balanced Propositional Matching (B): The balanced propositional matching index (Pirnay-Dummer & 
Ifenthaler, in press) is the quotient of propositional matching and concept matching. Especially when both indices 
are being interpreted, balanced propositional matching should be preferred over propositional matching. 
Propositional matching necessarily has its maximum in the value of concept matching. In order to balance this 
dependency of both indices, the balanced propositional matching index should be used instead of the concept and 
propositional matching. 

Results 

In testing our first research question, we wanted to understand the strengths and weaknesses of the applied 
methodologies (graphical and text re-representations). Therefore, we tested how good the internal consistencies 
within each methodology were and how much they differed. For our second research question, we tested the match 
between the structural and semantic indices for both methodologies. 

Consistency 

For both methodologies (graphical and text re-representations), the consistency was tested for all seven 
HIMATT measures (structural and semantic measures). As shown in Table 1, all structural errors and all semantic 
errors of variance are alike are alike except for the density of vertices measure. Based on this derivation, we keep the 
null hypothesis for the consistency: The graphical re -representations and the text-based re-representations have the 
same errors of variance. 



Consistency of graphical and text re-representations 
of structural and semantic indices 
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.81 


.79 


.71 


.6 

3 


.7 

4 


textual re- EV 


.26 


.22 


.22 


.22 


.32 


.3 


.3 


representation 


9 


6 


a 


.44 


.46 


.45 


.45 


.42 


.3 

9 


.4 

3 


Note. M = mean; EV= error of variance; a = internal consistency; R = surface 
matching; G = graphical matching; D = gamma matching; S = structural 
matching; C = concept matching; P = propositional matching; B = balanced 
propositional matching 



However, we found a difference between graphical and text-based re-representations (see Table 2). The 
text-based re -representations are by far more concise on the semantic level (concept matching, propositional 
matching, balanced propositional matching). Additionally, the text-based re -representations are significantly more 
complex than the graphical re-representations (graphical matching). Otherwise, the graphical re-representations are 
by far more concise concerning their density of vertices (gamma matching). Further, we found no significant 
differences between the text-based and graphical re-representations for the measures surface matching and structural 
matching (see Table 2). 
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Student t-Test; paired samples of 

Table 2: similarity measures (graphical & text- 

based re-representation) 



Measure 


t 


df 


P 


d 


surface matching 


.080 


187 


.936 


- 


graphical matching 


2.181 


187 


.030 


.196 


gamma matching 


-5.348 


187 


<.01 


.486 


structural matching 


-1.513 


187 


.132 


- 


concept matching 


4.797 


187 


<.01 


.380 


propositional matching 


5.294 


187 


<.01 


.433 


balanced propositional matching 


4.156 


187 


<.01 


.349 


Note, d = effect size 



Validity 

Following the second hypothesis, we tested correlatives between graphical and text-based re- 
representations (see Table 3). Although all as and means of similarities are statistically significant, the overall 
similarities are by far not sufficient to accept the alternative hypothesis. We therefore keep the null hypothesis: The 
indices of the graphical model representations and the indices from the text-based model re-representations correlate 
with r < .7. This means that text re-representations and graphical re-representations are attached to different 
constructs and therefore measure different things. 



Table 3: 



Correlatives of graphical and textual re-representations of structural 
and semantic indices 





R 


G 


D 


S 


C 


P 


B 


M 


.41 


.53 


.64 


.63 


.10 


.01 


.06 


EV 


.24 


.23 


.23 


.15 


.09 


.03 


.13 


a 


.29 


.35 


.39 


.39 


.09 


.01 


.05 



Note. M= mean; EV= error of variance; a = internal consistency; R = surface matching; G 
= graphical matching; D = gamma matching; S = structural matching; C = concept 
matching; P = propositional matching; B = balanced propositional matching 



Discussion 

The focus of our study was to compare the similarities and differences of the graphically and textually 
elicited knowledge structures and their semantics. Participants were asked to externalize their knowledge about the 
domain of research methods with two different elicitation methodologies - concept mapping and written text. 

For our analysis and comparisons of the externalized knowledge structures, we used the seven measures 
from the F1IMATT toolset (Pirnay-Dummer & Ifenthaler, in press; Pirnay-Dummer et ah, 2009). The automated 
quantitative analysis generated structural and semantic measures. The structural indicators were surface, graphical, 
gamma, and structural matching. The semantic indicators were concept, propositional, and balanced propositional 
matching. With the help of these indicators, we were able to precisely describe and track changes in students’ 
representations and compare the graphically and textually elicited knowledge. 

The fact that graphical notes and texts re -represent different things (even when used in the same task with 
the same participants) does not necessarily lead to the conclusion that either assessment is obsolete. In fact, it 
supports the model of information density vs. emphasis introduced above (see Figure 1). Further studies will have to 
be conducted to strengthen the theorem that re-representation values have different dimensions. But the results 
clearly show that concept mapping techniques and systematic text analysis can not be used as complements, e.g. in 
mixed-method attempts to triangulate theoretical constructs which are hard or impossible to observe directly. 

The graphical re-representation is structurally more concise, while the text-based re-representation is better 
on the semantic measures. At first sight, this may be contra-intuitive when we think about emphasis, because 
graphical re -representations have less semantic consistency. Flowever, this should in fact be expected: While the 
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whole model is constructed ( information density’) and little emphasis is conveyed in the textual re-representations, 
the graphical re -representations are idiosyncratic notes which happen to have a similar structure. Accordingly, there 
are two ways to interpret the similarities on the structural level: 

1. Concept maps are better for rebuilding a concise structure. 

2. Concept maps always have similar structures because their users only apply a certain range of 
structural patterns and therefore constrain themselves to the available patterns. 

The interpretations will have to be tested in a future study in which we will compare graphical structures 
from similar tasks but on completely different subject domains. If the structures turn out to be non-selective for any 
content or knowledge type, then we would have to favor the latter interpretation. If they are selective, the first 
interpretation has to be favored. 

The greater semantic similarities of the text re-representations are consistent with the model of information 
density’ vs. emphasis. Since the texts have a higher information density they contain more information and therefore 
more semantic terms match. Maybe the differences are a matter of objectivity - an assumption which can not be 
tested with the data at hand. This still leaves details about the differences between the aggregations open for 
investigation. 

Our future work will concentrate on larger experiments allowing us to analyze and compare graphical and 
text-based re -representations on completely different subject domains. These studies will help to strengthen the 
theorem that re-representation values have different dimensions, introduced in this study as information density vs. 
emphasis. 
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Reported Level of Prior Knowledge: A Study in Introductory Chemistry. 
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Abstract 

This exploratory study presents the results of the use of self-explanation strategies to increase learners’ 
performance with application on basic chemistry problems. The results of this randomized experiment confirm prior 
findings: learners can benefit from using self-explanation strategies, and the mediation effect between prior 
knowledge and self-explanation is suggestive of a synergic effect for learning. From an instructional design 
perspective, self-explanation can be adapted for effective tutoring sessions, review sessions, or short transfer 
problems, in both face-to-face and distance learning contexts. 

Introduction 

At the most fundamental level, when learning, to comprehend the material assigned students need to relate 
the new ideas to knowledge they already possess and to use that knowledge to form their own understanding and 
synthesize new knowledge. With many students struggling with concepts they encounter, especially when they are 
in their freshmen year, their instructors’ attempts to help these students often relies on threads of coherent reasoning 
constructed based on answers instructors received from them. Nevertheless, for objective reasons, an instructor is 
rarely available at the precise moment when a student needs help understanding a concept or working through a 
theory. Therefore, to ensure success, students need to employ a wide range of learning and reasoning strategies and 
rely on these strategies’ capabilities to help them make sense of the material they study. 

In this context, from the toolbox students have at their disposal, this study looked at self-explanation (e.g. 
Chi, Bassok, Lewis, Reinmann, & Glaser, 1989) as a possible strategy to support learners’ conceptual understanding 
and learning. Its objective was to find if this strategy could be used successfully to support learning in an 
introductory chemistry course in science education. Based on prior research findings (e.g. Bielaczyc, Pirolli, & 
Brown, 1995) which suggests a link between the effectiveness of this strategy and the domain-general and specific 
knowledge the learner has, this study looked at the effect prior knowledge, as reported by the learner, had on self- 
explanation strategies effectiveness. In addition, because this experiment was conducted online, using web-based 
technologies, its success suggests the potential of such strategies to be implemented in distance learning. 

Self-Explanation 

The term self-explanation or self-generated explanation (Chi, et al., 1989) refers to the explanation a 
learner generates on his or her own as opposed to the explanations provided by an external source (instructor, book, 
etc.). This is a domain independent strategy that can be used across domains and age groups with the capability to 
provide significant improvements in learners’ performance. Being a domain independent strategy, one important 
characteristic of self-explanation is that strategies based on it are reusable. That is, it provides these strategies with 
the capability of adaptability from one context to another and, more importantly, from one domain to another. 

Self-explanations are thought to be more effective than explanations provided by other sources because (a) 
they require students to actively elaborate their prior knowledge, thus triggering more constructive learning 
processes, (b) they are usually very well targeted to the learner’s specific problem, and (c) they are available at the 
precise time and place of the learner’s need. Reported gains in science education attributed to the use of self- 
explanation strategies are overwhelming, with self-explainers sometimes performing twice as well as the non-self- 
explainers (Kastens & Liben, 2007). 

The effect of self-explanations has been investigated by numerous researchers on subjects in a variety of 
conditions and tasks. For example, Chi & Bassok (1989) studied its influence on problem solving in physics while 
Pirolli & Recker (1994) looked at computer programming. Lin & Lehmann (1999) looked at experimental design 
while Kastens & Liben (2007) looked a map reading activities. Existing research documents the effectiveness of 
self-explanation based strategies across a variety of conditions. For example, Didirejean & Causinille-Marmeche 
(1997) studied its effectiveness when self-explanations are spoken aloud compared to when they just happen in 
one’s head and Aleven & Koedinger (2002) looked at it from the perspective of written versus typed expression 
media. It has also been documented that self-explanation based strategies help with (Aleven & Koedinger, 2002) and 
without (Chi, et al., 1989; Chi, De Leeuw, Chiu, & Lavancher, 1994) feedback on the correctness of the explanation. 

From a theoretical perspective, self-explanation was studied in the context of gap-filling (Chi & Bassok, 
1989; Lin & Lehmann, 1999; VanLehn, Jones, & Chi, 1992), mental model revision (DeLeeuw & Chi, 2003), 
conflict detection and resolution during knowledge integration (Chi, et al., 1994), thought organization (Lin & 
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Lehmann, 1999), and error detection and self-correction (Kastens & Liben, 2007). By construct, scholars have 
looked at schema formation and case -based reasoning (Didierjean & Cauzinille-Marmeche, 1997), analogical 
enhancement (Reinmann & Neubert, 2000), visual/verbal integration (Aleven & Koedinger, 2002), construction of 
new knowledge (Chi, et ah, 1989; DeLeeuw & Chi, 2003; Wong & Lawson, 2002), connection of principles to 
action (Lin & Lehmann, 1999), and situational model building (Kintsch, 1994). 

Looking at self-explanation as a domain independent strategy, Nathan et al. (1994) found that it works 
better for conceptual reasoning while providing only marginal advantage for procedural contexts. He also noted a 
decrease in performance when cognitive load is significant, situation that suggests the existence of a competition for 
cognitive resources. Bielaczyc et al. (1995) show that self-explanation is a strategy most high-performance students 
use, for example, when linking current concepts with prior knowledge. In his studies, the effectiveness of the 
strategy depends on the learners’ domain-general and domain-specific knowledge, the comprehensiveness of the 
problem being studies, as well as the state of the students’ evolving understanding. It has also been found that either 
guiding (e.g. Bielaczyc, et al., 1995) or prompting (Chi, et al., 1994) people to self-explain improves performance. 

For example, one area where self-explanation strategies can be successfully used is in tutoring. While some 
studies on self-explanation show that most learners do not spontaneously use self-explanation strategies (Conati & 
VanLehn, 2000) other studies suggest that learners seem to start self-explaining more when they are guided or 
prompted to do so (Bielaczyc, et al., 1995). Therefore one can consider that self-explanations strategies can be used 
to increase the effectiveness of tutoring or review sessions for problem solving through more intense conceptual 
engagement of the students. As an example, Chi (1996) uses successfully strategies involving self-explanation in 
tutoring students to solve mechanics problems. In this case, the tutor’s actions that prompted for the co-construction 
of knowledge (which includes self-explanation) proved to be the most beneficial in achieving deep learning, capable 
of overcoming misconceptions. That is, providing the tutees with support and opportunities to successfully construct 
answers on their own. 

The use of self-explanation strategies proved to be beneficial in online environments as well. One of the 
attempts in using an online environment to scaffold the use of self-explanation was undertaken by Atkinson and his 
colleagues (2003) who, in solving problems about probabilities, asked learners, aside from solving the problem, to 
specify the principle that applies to the particular problem they were working on. In this case a surprisingly simple 
procedure, prompt the participant to choose the principle underlying the problem, produced medium to strong effects 
on both near and far transfer. 

These are only two examples of how self-explanation has been used successfully to improve learners’ 
performance. As one can see, the interventions used in both cases are simple, nothing out of ordinary. What is 
important for success is the way they are used to support the learners’ effective use of self-explanation strategies. 

Our research was developed to target the same characteristics. We used prompt questions to elicit self-explanation at 
different stages in the learner’s path toward the solution. The challenge was to find how to construct these questions 
and how to time them to produce the expected effect, self-explanation, and not give away the solution to the 
problem. 

Instructional Context 

Incoming students often face difficulties in moving from learning strategies they used in high school to 
strategies that are effective for learning in college. These difficulties are more apparent in engineering and science 
programs where abstract knowledge and the skills needed to master them are critical to academic success. Among 
these difficulties, one major challenge students engaged in these transition processes face is their ability to build and 
successfully use deep reasoning skills. In the long run failure to address this challenge at institutional level results in 
poor student retention, which, in turn, contributes to a decrease in the attractiveness of engineering and science 
programs. To address this challenge, a Midwestern engineering college offers incoming students the option to enroll 
in a three -week summer program focused on the transition from high school to college instruction. This program is 
an excellent opportunity for incoming students to learn about coursework expectations in Mathematics, Chemistry, 
and English, as well as about campus life and community involvement. Continuously increasing enrollment in this 
summer program proves its positive impact for incoming students. In addition, the program provides incoming 
students with the opportunity to earn three hours course credit toward their academic degree. 

The focus of this study was the introductory chemistry module of the summer program presented above. In 
this module the students were engaged in a comprehensive study of the general principles of chemistry. 

This module emphasizes chemical nomenclature, periodicity of elements, chemical reactions and reaction 
stoichiometry, chemical bonding, and possible applications. During this module students took pop quizzes, work on 
homework assignments, take formal examination, and get grades for their performance. Participation in this research 
provided the students with the opportunity to earn extra points toward their final grade. While this group of future 
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engineering students was expected to have a certain level of prior knowledge in chemistry, to ensure equal 
opportunities for similar training and prior knowledge, this study was conducted at the end of the chemistry track. 

Research Questions 

The primary objective in this study was to find to what degree the use of self-explanation strategies 
improves students’ performance in basic chemistry problem solving. With prior knowledge singled out as affecting 
the effectiveness of self-explanation based strategies, a secondary goal in this research was to in which way prior 
knowledge, as reported by the students, affects self-explanation effectiveness. That is, the exploratory research 
questions were: 

(1) Does the use of self-explanation strategies increase learners’ performance in solving basic chemistry 

problems? 

(2) How does the use of self-explanation strategies relate to the learners’ perception of their own prior 

knowledge of the domain? 

Methods and Methodologies 

Participants 

A group of 80 incoming freshmen students enrolled in an intensive summer preparatory program were 
asked to solve a short chemistry problem and to answer a few demographics questions. At the time of the study all 
students were exposed to the chemistry track of the summer program, allowing them to address the task. 
Participation was voluntary and rewarded with extra points toward their chemistry section grade. A number of 52 
students completed all required tasks. No outliers were found and normality was a strong assumption for the group 
(one-sample Kolmogorov-Smirnov shows p> .05 for all continuous variables). 

Research design and procedure 

A two-group between subjects completely randomized experimental design was used (e.g. Keppel, 1991). 
That is, each participant was randomly assigned to only one of the two experimental groups, control or treatment. 
The experiment was delivered using a web-based research instrument designed and developed by the researchers 
together with a subject matter expert (SME), the instructor of the chemistry module. The application performed the 
random assignment of the participants on one of the two research conditions, control or treatment. Because once 
assigned to a group some students did not complete the entire set of tasks, the 52 students included in the final set 
were composed of 29 students assigned to the control group and 23 students assigned to the treatment group. 

The participants in both groups were asked to solve a short chemistry problem using the following 

scenario: 

Adrenaline, also referred to as epinephrine, is a sympathomimetic monoamine that is produced by the 
adrenal gland. This stress hormone, when secreted into the bloodstream, rapidly prepares the body for action in 
emergency situations. It increases heart rate and stroke volume, dilates the pupils, and constricts arterioles in the 
skin and gastrointestinal tract while dilating arterioles in skeletal muscles. It elevates the blood sugar level by 
increasing catabolism of glycogen to glucose in the liver, and at the same time begins the breakdown of lipids in fat 
cells. 

The students in the control group were asked to answer the following assessment question: 

Given that adrenaline contains 3 oxygen moles, 13 hydrogen moles, 9 carbon moles, and 1 nitrogen mole, 
which is its chemical formula? 

In contrast, the students in the treatment group were asked, before presenting them with the opportunity to 
answer the main assessment question, to answer the following three guiding questions: 

(1 ) What type of chemical compound is adrenaline ? 

(2) How does belonging to this type of chemical compounds influence the way chemical formulae are written? 

(3) How do you write the formula for this type of compound? 

These guiding questions, developed in collaboration with the subject matter expert, aimed at engaging 
students in self-explanation behavior. The nature of these questions emulated the causal structures an expert would 
activate when answering the main question. 

In the next step, the participants in both the control and treatment groups were asked to indicate how 
confident they are in the correctness of their answer for the main assessment question and then to estimate their own 
level of chemistry knowledge. Finally, the participants were asked several questions related to their individual 
learning characteristics and to provide demographic information. 

Dependent variables 

Two categories of measures were used in this study. The first category included measures of students’ 
performance and the associated confidence that the answer they provided was correct. The performance was 
measured as a binary outcome 0 = wrong answer/1 = correct answer. In addition participants’ confidence in the 
accuracy of the answer was generated as a continuous variable, varying from 0 for low confidence to 100 for high 
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confidence. To account for the complexity of students’ performance the binary outcome was complemented with an 
adjusted performance variable computed as follows: 

Adjusted performance = (-1) * Confidence for wrong answer+ 100 and 

Adjusted performance = Confidence for the correct answer + 100 

The resulted dependent variable named adjusted performance, ranges from (0) for wrong answer with high 
confidence that the answer is right to (200) for correct answer with high confidence that the answer is right. 

The second group of dependent variables was deployed to test if there is a significant difference between 
the control and treatment group in terms of individual characteristics associated with learning in this program. The 
two dependent variables in this second group were motivation and academic efficacy, and were adapted from 
Midgley et al. (1998; 2000). These constructs used a five point Likert scale and the final value resulted as a mean of 
the items in each scale. The value of these variables varied from 1 for low to 5 for high values of the respective 
construct. 

Independent variables 

The first independent variable was the group to which the participants belonged, a categorical variable with 
two levels based on allocated experimental condition, control and respectively treatment. The second independent 
variable, self-reported familiarity with the field of chemistry, was collected as a continuous variable varying from 0 
for low familiarity to 100 for high familiarity with the field of chemistry. This second independent variable collected 
to test if prior knowledge behaves as a moderator, as suggested by the existing research literature (Bielaczyc, et al., 
1995) where prior knowledge is viewed as an important ingredient for effectiveness of self-explanation strategies. 
Table 1 summarizes the statistical characteristics of continuous variables used in this study. 



Table 1 

Means, Standard Deviations, and Pearson Correlations for continuous variables 





M 


SD 


1 


2 


3 


4 


1 . Adjusted performance 


111.22 


73.75 


- 


.29* 


.29* 


.01 


2. Academic Efficacy 


4.12 


.58 




- 


.52** 


.14 


3. Motivation Goal Orientation 


3.93 


.64 






- 


-.02 


4. Self-reported knowledge 


44.78 


27.01 








- 



Note: *p < .05; **p < .01 



Results and Interpretation 

One-way ANOVA with one between-groups factor was used to test the homogeneity of the two 
experimental groups. No statistically significant differences were found between students’ learning characteristics in 
the two experimental groups as measured by academic efficacy, F(l,49) = 1.16, p = .29, motivation goal 
orientation, F(l,49) = .63, p = .43, and self-reported knowledge of chemistry, F(l,50) = 2.54, p = .12 . Therefore, 
the two groups randomly created for this experiment had similar learning characteristics. 

To analyze if the use of self-explanation strategies increased learners’ performance on basic chemistry 
problems we first use a chi-square test with binary performance as dependent variable. We found that significantly 
more students in the treatment group (using self-explanation strategies) answered correctly to the chemistry problem 
when compared to students in the control group, yj (1, N=52) = 6.32, p < .05. 

Linear regression was used to analyze how the use of self-explanation strategies relates to the learners’ 
perception of their own prior knowledge of the domain. The adjusted score was the dependent variable, self- 
explanation treatment was the categorical variable treated as dummy variable with 1 for the treatment group and 0 
for the control group, while the self-reported knowledge of chemistry was the moderating variable. The focus was 
on the moderating effect self-reported knowledge of chemistry might have on the relationship between the adjusted 
score and the self-explanation treatment. The reason for choosing the adjusted score for the dependent variable in 
this analysis was its ability to enhance the value of a binary right/wrong answer to account for the students’ 
confidence in the validity of the provided answer and by this means, reflect the strength of their mental model. 

The bivariate correlations revealed one significant predictor for the adjusted score, the treatment (r = 0.37) 
significant at p < 0.05. In a first step the adjusted score was then regressed on the treatment and the self-reported 
chemistry knowledge. The resulted equation accounted for 15% of the variance in the final exam score, F(2,49) = 
4.16, p < 0.05. Only treatment displayed a significant beta weight of (3 = .39 (p < .05). 

In a second step, the interaction between the treatment and the self-reported chemistry knowledge score 
was introduced as predictor mean centered values for self-reported chemistry knowledge were used in the analysis. 
In this second step, the interaction term between attendance behavior and homework explained a significant increase 
in final exam score, AR 2 = 0.16, F(l,48) = 1 1.06, p < 0.05. Thus, the self-reported chemistry knowledge was a 
significant moderator of the relationship between treatment and adjusted score (Table 2). 



97 




Table 2 

Summary of Regression Analysis 



Predictor 


Beta Weights 

Beta 


t 


Treatment (A) 


.41 


3.35** 


Self-reported knowledge (B) 


-.26 


-1.59 


A * B 


.54 


3.33** 


Model Summary 


= 0.31, p < 0.01 





Note: N = 52. **p < .01 



The unstandardized simple slope of adjusted score for the treatment group was + 1.55, and the 
unstandardized simple slope for the control group was - 0.71 (Aiken & West, 1991). The simple slope analysis 
indicated that the positive slope for the treatment group was statistically significant, t(48) = 3.01 > t cr (40) = 2.707, p 
< 0.01, while the negative slope of the control group was not statistically significant (Figure 1). 
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Figure 1. Moderating effect of self-reported prior knowledge on the use of self-explanation strategies 

Discussions and Implications 

The results of this study confirms, on one hand, that learners can benefit from using self-explanation 
strategies in learning chemistry and, on the other hand, the mediation effect of prior knowledge on the outcomes of 
using self-explanation based strategies is suggestive for a synergic effect on the subjects’ ability to solve chemistry 
problems. The interaction between prior knowledge and the use of self-explanation based strategies seems to be two- 
folded. On one side, the higher the self-reported knowledge is, the more powerful the effect of self-explanation 
strategy is, in agreement with the behavior suggested by prior research. Also, using strategies based on self- 
explanation tends to help students make better use of their prior knowledge of the domain in their current activities 
by “forcing” them to consider it when working on current tasks. On the other side, for these strategies to be 
effective, a certain level of prior knowledge (threshold) seems to be needed. That is, if prior knowledge is too low, 
the strategy seems to be acting in the opposite direction, effectively diminishing student performance in solving 
chemistry problems. Present research was not designed to study prior knowledge but only to account for it. It is for 
future research to investigate the nature and threshold level of prior knowledge that makes self-explanation based 
strategies effective. 

From a developmental perspective, these findings suggest that a layered deployment of self-explanation 
based elicitation strategies has the potential to increase the effectiveness of the knowledge construction processes. 

Its purpose should be to provide students with this tool and to train them in using it. That is, the first stage could 
consist of constant use of self-explanation elicitation questions to train the students in using these strategies, 
followed by a second stage, where the use of these questions would be faded out allowing students to further adapt 
the use of these strategies to their own needs and characteristics. 
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From an instructional design perspective this strategy can be adapted to increase the effectiveness of 
tutoring and review sessions or short transfer problems through more intense cognitive engagement of the students 
during the various learning activities. The fact that this experiment was conducted entirely using web-based 
technologies can be an indication that methodologies based on self-explanation elicitation strategies could be readily 
embeddable in online learning environments. 

As for the future, research should look for example at how various characteristics of the elicitation 
questions, such as focus, atomicity, concept vs. process targeting, etc. influence learners’ performance. For example, 
in science and engineering, where the question asking strategy is frequently used in classroom-based activities, 
future research could look at the effect of generic vs. specific prompt questions or at how timing of the questions 
could affect the strategy’s effectiveness. 
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Analysis of peer-scaffolding patterns in four phases of problem-solving 

in web-based instruction 
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Introduction 

Problem-solving is regarded as one of the most important competency in everyday life and many professional 
contexts. Thus, in recent years, there has been an increased emphasis on improving students’ problem-solving 
abilities (Phye, 2001). One of the most effective approaches for students to enhance their problem-solving 
abilities is the utilization of web-based environments that allow them to provide peer-scaffoldings to solve a 
given set of problems. Web-based environments allow learners to develop their problem-solving abilities, build 
knowledge, and interact with peers through a web-based learning system (Chou, & Tsai, 2002; Lee, 2004). On 
the web, students may liberally join discussions to get information related to problem-solving, or provide peers 
with support. 

It is difficult for learners to solve problems successfully without an instructor’s or peer’s support in a 
web-based environment. This is because some obstacles, such as the learner’s feeling of isolation and the still of 
communication remain (Sharma & Hannafin, 2007). To overcome these obstacles, scaffolding may play an 
important role, as experts provide support in these environments and are involved in giving appropriate help. 

Although several studies have been conducted on the various scaffoldings in problem-solving situations, it is 
really hard to find out the researches on peer-scaffoldings which can support to the learners’ problem solving 
processes. It is therefore important to understand peer-scaffolding patterns for collaborative problem-solving in 
web-based instruction environments. The purposes of this study are to explore the types of peer-scaffolding that 
occur during the problem-solving processes in web-based instruction, as well as to identify what patterns of 
peer-scaffolding for problem-solving are used to effectively solve problems from the early phase to the last 
phase. 

Theoretical Background 

A. Concepts and types of scaffoldings 

The concept of scaffolding originated in Vygotsky's socio-cultural theory. Vygotsky (1978) described learning 
that occurs in the zone of proximal development (ZPD) -the space in which a child can do something, such as 
complete a task or attain a specific goal with an expert’s or more capable peer’s support. 

During the last decade, there have been several studies on various scaffolding types, such as explanation, 
feedback, direction maintenance, demonstration, and modeling (Bull, Shuler, Overton, Kimabll, Boykin, & 
Griffin, 1999; Roehler, & Caltlon, 1997; Woods, Bruner, & Ross, 1976). These types of scaffolding have been 
found effective in fostering problem comprehension, problem-solving, and reflective thinking. In addition, many 
researchers emphasize how scaffolding can be used in various ways, such as providing explanations or cues and 
inviting student participation (Hogan & Pressley, 1997; Kim, 1997; Winnips, & McLoughlin, 2000). 

B. Four phases of problem-solving 

According to many scholars, like Gagne (1985), and Smith and Ragan (1999), students generally go through 
similar problem-solving processes regardless of a problem’s characteristics. Newell and Simon (1972) explained 
that the general problem-solver specifies two sets of problem-solving processes: understanding processes and 
searching processes. Gick (1986) synthesized various other problem-solving models and proposed a simplified 
model of the problem-solving process: constructing a problem representation, searching for solutions, 
implementing, and monitoring solutions. Noh and colleagues (2001) proposed that the four stages of the 
problem-solving strategy are understanding, planning, solving, and checking while investigating the influences 
of an instructional method related to problem-solving. Ge and Land (2003) studied the effects of question 
prompts and peer interactions in scaffolding learners’ problem-solving processes. In their study, they divided 
problem-solving into four processes: problem representation, developing solutions, making justification for 
generating or selecting solutions, and monitoring and evaluating the problem space and solutions. Considering 
those phases of problem-solving processes that all researchers suggested, the process can be categorized into 
four phases: understanding, planning, solving, and reviewing. 

C. Relationships between scaffolding and problem-solving 

Scaffolding has been found effective in fostering problem-solving (Scardamalia, Bereiter, & Steinbach, 1984). 
Hogan and Pressley (1997) emphasize that scaffoldings are needed for students to build powerful thinking 
strategies for learner inquiries or solving a problem. Choi and colleagues (2005) report that, through scaffolded 
instruction, learners take more responsibility for their own learning and become more independent learners. In 
other words, learners’ responsibility and independence, both necessary for successful problem-solving, can be 
built up by providing scaffolding. Scaffolding is definitely necessary for students to solve problems successfully 
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because scaffolding helps learners to understand the relationships between what they already knew and the new 
information needed to solve a problem. 

Methods 

A. Participants 

Six participants were involved in solving the given problems as a group. Solving these problems, they shared 
peer-scaffoldings collaboratively. All of the participants had worked in middle-, upper-, and higher-positions in 
different fields of administrative divisions, such as research affairs, student affairs, and academic affairs at the 
three universities in Seoul, Korea. In addition, they had worked for more than 10 years in their professional 
fields. The age range of the six participations, four males and two females, was 40 to 50. 

B. Problems as a learning task 

Participants worked collaboratively in a group and were required to solve problems in a learning task. 

Problems in the learning task were: ( 1) to identify topics that will be helpful for solving problems that they may 
come across in their working fields, (2) to search for data and information that is related to problem solving, and 
(3) to delineate the results of the problem-solving process in an approximately 50-page report. 

C. Course setting 

This course consisted of 15 weeks of a ‘advanced leadership training program’, in the spring semester of 2008. 
They were instructed to collaboratively solve the given problems as a learning task with their peers. This study 
was conducted in a web-based environment. The participants had to discuss, in an online community, at least 15 
hours over the 15 week period in order to solve the problems as a learning task. 

D. Procedures of research and data analysis 

Data from collaborative on-line discussions were collected through the website for the peer-scaffolding 
analysis. The website was run in a cyber teaching learning class. There were two steps in analyzing the data, as 
two steps will be discussed below. 

1. Four phases of problem-solving for analyzing the data 

The four phases of problem-solving as shown in Table 1 are understanding, planning, solving, and reviewing. 
Problem-solving activities in each phase are presented in Table 1. Data were classified into the four phases of 
problem-solving accordingly. 



<Table 1> Four phases of problem-solving and problem-solving activities in each phase 



Four phases of Problem-solving 


Problem-solving activities 


Understanding 


find the given information and goal 
confirm the topic to write the report 


Planning 


identify additional knowledge to solve the problems 
select and organize the contents 
discover strategies to write the report 


Solving 


collect the information to write the report 
write the report collaboratively 


Reviewing 


check the solving processes 

review the things to learn 

reflect on the appropriateness of solutions 



2. Coding scheme to analyze peer-scaffoldings according to the four phases of problem-solving 

In order to analyze the patterns of peer-scaffolding during the collaborative problem-solving activities, a 
coding scheme for analyzing peer-scaffoldings was developed and elaborated. To verify the coding scheme, the 
tool that Jang (2005) applied in her study was revised and applied in the present study. Using this tool, the 
coding scheme for analyzing the peer-scaffoldings in Table 2 was validated by experts (1 professor and 3 
doctoral candidates) in the instructional psychology field. These experts verified the coding scheme by using a 5 
point Likert scale (5=Fully verified, l=no verified) to rate explicability, usability, validity, comprehensibility, 
and generality with respect to dimensions of peer-scaffolding, types of peer-scaffolding, definitions of peer- 
scaffolding, and examples of peer-scaffolding. The results of experts’ verification were shown the average 
points of 3.92 out of 5.0. This result confirms that the coding scheme for analyzing peer-scaffoldings is fairy 
valid. 
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< Table 2 > Coding scheme for analyzing the peer-scaffoldings 



Dimensions 
of peer- 
scaffolding 


Types of 
peer- 

scaffolding 


Definition of peer-scaffolding 


Examples 


Strategy 


maintaining 

direction 


Statements about maintaining 
attention on a goal by requesting 
clarification. 


For completing this task, why don’t we consider 
not only public control universities, but also 
private control universities as the best case? 


assigning 

role-taking 


Statement about something to do 
to solve problem or task 


For gathering meaningful information or materials, 
we need to assign individual roles. I’d like to 
suggest that Mrs. Jang search on this part because 
Mrs. Jang is an expert on this. 


Content 


offering cue 


Statements related to problem 
area to focus on completing the 
tasks. 


When I was a graduate student in Japan, I carried 
out a survey similar to this study. 

That survey may be helpful for our task. So I 
attached the survey report with reference to our 
work. 


Offering 

opinion 


Statements describing one’s 
thoughts, such as alternative 
plans to facilitate problem- 
solving, or tactics to discover a 
solution. 


According to my opinion on administrative 
services to faculty and students in universities, the 
professionalism of the staff is a critical factor, I 
think. 


offering 

explanation 


Statements adjusted to fit the 
learner’s enhancing 
understanding about what is 
being learned, why and when it 
is used, and how it is used. 


I think it would be better to understand ERIC, 
which is explained in the blue ocean strategies. 
ERIC stands for Eliminate, Reduce, Increase, and 
Create. Eliminate means that consumers are. . .. 


offering 

feedback 


Statements of evaluation of 
peers’ work and pointing out the 
distinction between the learner’s 
performance and the ideal. 


After reading the current survey, we need to add 
content related to staff promotion to what we’re 
about to write. 


Affection 


offering 

praise 


Statements that are about 
encouragement or inspiring peers 
to further effort, and comments 
on peers’ good work. 


Mrs. Jang! ERIC is really important content for 
completing the task. You provided such 
appropriate contents to solve this problem. Thank 
you!! 


Inviting 

participation 


Statements that asked learners to 
join in 

problem-solving activities. 


Mrs. Jang, I want to get your opinion. Please read 
the contents of the attached file and give me your 
opinion or feedback. 



Two coders analyzed independently peer-scaffolding messages with reference to the coding scheme validated 
by experts. The inter-rater reliability between the two coders was calculated for 75% of the data. The inter-coder 
reliability (agreement) in each analysis was 73%. 



Results and Discussion 

A. The number of peer-scaffolding messages, and ratio according to problem-solving phases 

According to our coding scheme, we analyzed all the messages that were discussed among peers in the online 
community. The total number of messages from learners was 77 postings on the bulletin board. More 
specifically, the number of initiating messages was 36, and the number of reply messages was 41. Most 
messages consisted of approximately 10-20 sentences. In the content analysis, the unit of analysis was the 
semantic unit. Each message segment was analyzed in light of the eight scaffolding coding scheme categories 
(See Table 3). 
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<Table 3> The numbers of peer-scaffolding messages and ratio by problem-solving phases 



' — — Eroblem-solving phases 
^Scaffolding — — — 
Dimension types 


Understanding 

phase 


Planning 

phase 


Solving 

phase 


Reviewing 

phase 


Total( %) 




maintaining direction 


6 


1 


6 


1 


14(8) 


strategy 


assigning role-taking 


0 


3 


8 


5 


16(9) 




sub total 


6 


4 


14 


6 


30 (18) 




offering cue 


4 


3 


21 


0 


28 (16) 




offering opinion 


7 


0 


9 


0 


16(9) 


content 


offering explanation 


8 


1 


8 


6 


23 (14) 




offering feedback 


8 


0 


1 


2 


11 (6) 




sub total 


27 


4 


39 


8 


78 (46) 




offering praise 


5 


2 


15 


13 


35 (20) 


affection 


inviting participation 


9 


3 


13 


2 


27 (16) 




sub total 


14 


5 


28 


15 


62 (36) 


Total (%) 


47(28) 


13(7) 


81(48) 


29(17) 


170(100) 



With respect to the problem-solving phases, the number of peer-scaffoldings was 47 (28%) in the 
understanding phase, 1 3 (7%) in the planning phase, 8 1 (48%) in the solving phase, and 29 ( 1 7%) in the 
reviewing phase. In addition, in terms of peer-scaffolding dimensions, the number of peer-scaffoldings was 30 
(18%) in the strategy dimension, 78 (46%) in the content dimension, and 62 (36%) in the affection dimension, 
respectively (See Table 3). 

B. Patterns of peer-scaffoldings in each dimension of the four phases of problem-solving 

All the messages were analyzed to reveal the patterns of peer-scaffoldings in each dimension of the four phases 
of problem-solving. As shown in Figure 1 , peer-scaffoldings in the content dimension represented a higher ratio 
of peer-scaffoldings in the understanding phase and solving phases than did those in the strategy dimension or 
affection dimension. Further, a chi-square analysis on peer-scaffolding patters according to these dimensions 
demonstrated that there was a statistically non-significant difference between the strategy, content, and affection 
dimensions (>7= 8.805, p>.05) (see Tables 4, 5). 

More specifically, peer-scaffoldings in the strategy dimension were 18% of all peer-scaffoldings across all 
three dimensions. The strategy dimension includes the ‘maintaining direction’ scaffolding and the ‘assigning 
role-taking’ scaffolding (see Figure 1). The reason for this pattern is that the participants tended to keep the 
strategies that they had in their mind as the most effective and efficient strategies. This means that even though 
they know various strategies, they use more familiar strategies when solving a problem without considering the 
context. In addition, they could easily have had similar ideas when they made plans to find a solution because all 
of the participants had similar work experiences in the administrative field. 

The peer-scaffoldings in the content dimension were 46% of all peer-scaffoldings across all three dimensions, 
and content dimension includes the four peer-scaffoldings of ‘offering cue,’ ‘offering opinion,’ ‘offering 
explanation,’ and ‘offering feedback’ (see Figure 1). This result suggests that participants have sufficient expert 
knowledge related to problem-solving, especially as participants were sufficiently experienced for a long time in 
their professional job areas. 

The peer-scaffoldings in the affection dimension accounted for 36% of all peer-scaffoldings across all three 
dimensions, and the affection dimension includes the ‘offering praise’ and ‘inviting participation’ scaffoldings. 
This result shows that participants recognized that peer-scaffoldings in the affection dimension were more 
important than scaffoldings in the strategy dimensions. 

Comparing the eight types of scaffoldings, however, we see that the number of messages in the ‘offering 
praise’ scaffolding was the highest among all types of scaffolding. This implies that adults consider a permissive 
atmosphere to be an important factor for successfully solving problems (see Figure 1). 
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<Figure 1> Distribution of peer-scaffoldings in all problem-solving processes 



<Table 4> Dimension * problem-solving phase Cross-tabulation 









problem-solving phase 


Total 








Under- 


Planning 


Solving 


Reviewin 


Dimensio 


Strategy 


Message No. 


6 


4 


14 


6 


30 


n 




%within 


20.0 


13.8 


46.7 


20.0 


100.0 




% within nhase 


12.8 


30.8 


17.3 


20.7 


17.6 






% of total 


3.5 


2.4 


8.2 


3.5 


17.6 




Content 


Message No. 


27 


4 


39 


8 


78 






%within 

dimension 


34.6 


5.1 


50.0 


10.3 


100.0 






% within phase 


57.4 


30.8 


48.1 


27.6 


45.9 






% of total 


15.9 


2.4 


22.9 


4.7 


45.9 




Affection 


Message No. 


14 


5 


28 


15 


62 






%within 

dimension 


22.6 


8.1 


45.2 


24.2 


100.0 






% within phase 


29.8 


38.5 


34.6 


51.7 


36.5 






% of total 


8.2 


2.9 


16.5 


oo 

OO 


36.5 


Total 




Message No. 


47 


13 


81 


29 


170 






%within 

dimension 


27.6 


7.6 


47.6 


17.1 


100.0 






% within phase 


100.0 


100.0 


100.0 


100.0 


100.0 






% of total 


27.6 


7.6 


47.6 


17.1 


100.0% 



<Table 5> %2 test on the patterns of peer-scaffolding according to dimensions 





value 


df 


Asymp. Sig. (2-sided) 


Pearson Chi-Square 


8.805 a 


6 


.185 


Likelihood Ratio 


8.795 


6 


.185 


N of Valid Cases 


170 







a. Cells (16.7%) have expected counted less than 5. The minimum expected counted is 2.29. 



C. Patterns of peer-scaffolding according to each peer-scaffolding dimension 
1. Patterns of peer-scaffoldings in the strategy dimension 

A chi-square test on patterns of peer-scaffolding by strategy dimension demonstrated that there was a 
statistically significant difference between the strategy, content, and affection dimensions (7 = 9,863, p<-05) 
(see Tables 6, 7). Despite the statistical significance of these results, it should be careful about accepting the 
significance, as 9 out of 16 cells had expected counts under five. More specifically, considering the patterns of 
peer-scaffolding in the strategy dimension (see Figure 2), the ‘maintaining direction’ scaffolding represents a 
higher ratio than the ‘assigning role-taking’ scaffolding in the understanding phase. This result suggests that 
participants discussed ‘what learner should think of completing the task,’ and ‘what goals and directions are 
needed to solve the problem.’ After understanding problems as a task, the ‘assigning role-taking’ scaffolding 
was higher frequency scaffolding than ‘maintaining direction’ scaffolding from the planning phase to the 
reviewing phase. The reason is that, after understanding the problems, it was crucial for participants to commit 
to roles for problem-solving and become involved in the problem-solving activities. 
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<Figure 2> Patterns of peer-scaffoldings in strategy dimension 



<Table 6> strategy * problem-solving phase Cross-tabulation 









nroblem-solving nhase 










Under- 


Planning 


solving 


reviewing 


Total 


















strategy 


maintaining 


Message No. 


6 


1 


6 


i 


14 




direction 


%within 


42.9 


7.1 


42.9 


7.1 


100.0 




% within 


100.0 


25.0 


42.9 


16.7 


46.7 






% of total 


20.0 


3.3 


20.0 


3.3 


46.7 




assigning 


Message No. 


0 


3 


8 


5 


16 




role-taking 


%within 


.0 


18.8 


50.0 


31.3 


400.0 




% within 


.0 


75.0 


57.1 


83.3 


5.3.3 






% of total 


.0 


10.0 


26.7 


16.7 


53.3 


Total 




Message No. 


6 


4 


14 


6 


30 






%within 

dimension 


20.0 


13.3 


46.7 


20.0 


100.0 






% within 

phase 
% of total 


100.0 


100.0 


100.0 


100.0 


100.0 






20.0 


13.3 


46.7 


20.0 


100.0 



<Table 7> yl test on the patterns of peer-scaffolding in the strategy dimension 





Value 


df 


Asymp. Sig. (2-sided) 


Pearson Chi-Square 


9.863 a 


3 


.020 


Likelihood Ratio 


12.429 


3 


.006 


N of Valid Cases 


30 







a. 6Cells (75.0%) have expected counted less than 5. The minimum expected counted is 1.87. 



2. Patterns of peer-scaffolding in the content dimension 

A chi-square test on the patterns of peer-scaffolding in the content dimension showed that there was a 
statistically significant difference among the ‘offering cue’, ‘offering opinion’, ‘offering explanation’, and 
‘offering feedback’ (X = 30.639, p<-05) (see Table 8, 9). Again, despite the statistical significance of this result, 
it should be careful in accepting this, as 9 of 16 cells had expected counts lower than five. More specifically, the 
‘offering cue’ scaffolding represents a higher ratio than do the other peer-scaffoldings in the planning phase and 
in the solving phase, as participants recognized that their peers had high levels of problem-solving abilities and 
had accumulated prior knowledge (See Figure 3). This result means that cues provided by adults facilitated 
learners’ active problem-solving. 




— offering Cue 

offeringopinion 

offering explanation 

— - — offeringfeedback 



<Figure 3> Patterns of peer-scaffoldings in the content dimension 
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<Table 8> content * problem-solving phase Cross-tabulation 





nroblem-solving nhase 


total 


Under- 

standing 


Planning 


solving 


reviewin 

g 


conte offering Message No. 


4 


3 


21 


0 


28 


nt cue %within 


14.3 


10.7 


75.0 


.0 


100.0 


% within nhase 


14.8 


75.0 


5.3.8 


.0 


35.9 


% of total 


5.1 


3.8 


26.9 


.0 


35.9 


offering Message No. 


8 


1 


8 


6 


23 


explanati %within 


34.8 


4.3 


34.8 


26.1 


100.0 


% within nhase 


29.6 


25.0 


20.5 


75.0 


29.5 


on % of total 


10.3 


1.3 


10.3 


7.7 


29.5 


offering Message No. 


8 


0 


1 


2 


11 


feedback %within 


72.7 


.0 


9.1 


18.2 


100.0 


% within phase 


29.6 


.0 


2.6 


25.0 


14.1 


% of total 


10.3 


.0 


1.3 


2.6 


14.1 


offering Message No. 


7 




9 


0 


16 


opinion %within 


43.8 


.0 


56.3 


.0 


100.0 


% within nhase 


25.9 


.0 


23.1 


.0 


20.5 


% of total 


9.0 


.0 


11.5 


.0 


20.5 


Total Message No. 


27 


4 


39 


8 


78 


%within 














34.6 


5.1 


50.0 


10.3 


100.0 


dimension 












% within phase 


100.0 


100.0 


100.0 


100.0 


100.0 


% of total 


34.6 


5.1 


50.0 


10.3 


100.0 



<Table 9> yl test on the patterns of peer-scaffoldings in the content dimension 





value 


df 


Asymp. Sig. (2-sided) 


Pearson Chi-Square 


30.639 3 


9 


.000 


Likelihood Ratio 


35.671 


9 


.000 


N of Valid Cases 


78 







a. 9 Cells (56.3%) have expected counted less than 5. The minimum expected counted is .56. 



3. Patterns of peer-scaffolding in the affection dimension 

A chi-square test on patterns of peer-scaffolding in the affection dimension demonstrated a statistically 
significant difference between the ‘offering praise’ scaffolding and the ‘inviting participant’ scaffolding (X‘= 
8.664, p< .05) (see Tables 10, 11). More specifically, the ‘inviting participation’ scaffolding represented a 
higher ratio than did the ‘offering praise’ scaffolding in the understanding phase. 

The understanding phase is an early part of the problem solving process, and participants are not yet intimate 
friends each other in this early phase. Accordingly, participants did not actively interact with their peers to solve 
the problems. So, in the understanding phase, they provided ‘inviting participation’ scaffolding more than 
‘offering praise’ scaffolding. On the other hand, in the later reviewing phase, the ‘offering praise’ scaffolding 
represented a higher ratio than did the ‘inviting participation' scaffolding (see Figure 4). The reason is that when 
participants reviewed the problem-solving process, they were more likely to offer praise to peers so that they 
were able to actively engage in problem-solving and complete the task. 




<Figure 4> Patterns of peer-scaffolding in the affection dimension 
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<Table 10> affection * problem-solving phase Cross-tabulation 





nroblem-solvine nhase 


total 


Under- 

standing 


Planning 


solving 


reviewing 


Affecti inviting Count 


9 


3 


13 


2 


27 


on Participati % within 


33.3% 


11.1% 


48.1% 


7.4% 


100.0% 


% within nhase 


64.3% 


60.0% 


46.4% 


1.3.3% 


4.3.5% 


0n % of total 


14.5% 


4.8% 


21.0% 


3.2% 


43.5% 


offering Count 


5 


2 


15 


13 


35 


Praise %within 


14.3% 


5.7% 


42.9% 


37.1 % 


100.0% 


% within nhase 


35.7% 


40.0% 


5.3.6% 


86.7% 


56.5% 


% of total 


8.1% 


3.2% 


24.2% 


21.0% 


56.5% 


Total Count 


14 


5 


28 


15 


62 


%within 












dimension 


22.6% 


8.1% 


45.2% 


24.2% 


100.0% 


% within phase 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


% of total 


22.6% 


8.1% 


45.2% 


24.2% 


100.0% 



<Table 1 1> yl test on the patterns of peer-scaffolding in the affection dimension 





value 


df 


Asymp. Sig. (2-sided) 


Pearson Chi-Square 


8.664 a 


3 


.034 


Likelihood Ratio 


9.482 


3 


.024 


N of Valid Cases 


62 







a. 2 Cells (25.0%) have expected counted less than 5. The minimum expected counted is 2.18. 



Conclusion and Implications 

Various peer-scaffoldings are critical elements for effective problem-solving. But little research exists on what 
patterns and types of peer-scaffolding are practiced during group problem- solving, and what types of peer- 
scaffolding occur during the four problem-solving phases. 

The purposes of this study were to analyze the types of peer-scaffolding messages during on-line problem- 
solving, and to investigate patterns of peer-scaffolding as effective problem-solving strategies in the four phases 
of problem-solving. 

The major findings of this study on the patterns of peer-scaffolding are as follows: First, peer-scaffoldings in 
the strategy dimension accounted for only 18% of all peer-scaffoldings across all three dimensions. According 
to the chi-square value, this resulted from a statistically significant difference though it should be careful to 
interpret this result given expected counts under five messages in the chi-square test. More specifically, 
considering patterns of peer-scaffolding in the strategy dimension, the ‘maintaining direction’ scaffolding 
represented a higher ratio than did the ‘assigning role-taking’ scaffolding in the understanding phase. After 
understanding phase, the ‘assigning role-taking’ scaffolding was higher frequency scaffolding than ‘maintaining 
direction’ scaffolding from the planning phase to the reviewing phase. In this sense, instructor needs to 
encourage learners to use various strategies that they have to keep a direction in understanding phase. And after 
understanding phase, instructor should guide for students to assign role to team members appropriately. 

The chi-square analysis on the pattern of peer-scaffolding in the content dimension was significant different 
among the ‘strategy’ dimension, ‘content’ dimension, and ‘affection’ dimension. But it should be interpreted 
with caution since the number of messages was under five. Despite this caution, this result implies that it is 
important to check the extent of the learners’ prior knowledge and adapt the problems’ difficulty level 
accordingly. In this sense, tutors should encourage adult learners to make use of prior knowledge in the 
problem-solving process. More concretely, considering patterns of peer-scaffolding in the content dimension, 
the ‘offering cue’ scaffolding was a higher proportion here than were other peer-scaffoldings in the solving 
phase. This result implies that, when instructor provides scaffolding to learners, instructor should provide the 
‘offering cue’ scaffolding instead of the ‘offering explanation’ or the ‘offering feedback’ scaffoldings in the 
planning phase and in the solving phase. In addition, the ‘offering cue’ scaffolding should be considered a 
critical factor in designing problem-solving instruction. 

Third, in the affection dimension, the ‘inviting participation’ scaffolding represented a higher ratio than did 
other peer-scaffoldings in the understanding phase. On the other hand, in the reviewing phase, the ‘offering 
praise’ scaffolding represented a higher ratio than the ‘inviting participation' scaffolding. These results imply 
that, when instructors provide peer-scaffoldings in affection dimension, which are ‘offering praise’ scaffolding 
and ‘inviting participation’ scaffolding, they should consider the phase of problem solving carefully. In other 
words, instructor needs to guide for learner to invite others’ participation in understanding phase and to offer 
praise to others in reviewing phase. 
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In addition, comparing all different types of scaffoldings, the ‘offering praise’ scaffolding is the highest ratio 
among all types of peer-scaffoldings. This result implies that, when we design collaborative problem-solving 
instruction in a web-based environment, it must be considered the ‘offering praise’ scaffolding as critical 
component. In addition, instructors should be guided to offer more praise to their adult learners appropriately. 
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Using CTL-based Online Discussion Strategies to facilitate Higher Level 

Learning 

Li Jin, Aubteen Darabi, and Thomas Cornille 

Background 

Online discussion has become a major instructional component of face-to-face, online and blended 
instruction. It has been found to enhance students’ learning by supporting social learning (Wells, 1992), mindfulness 
and reflection (Hiltz, 1994; Poole, 2000), and idea sharing and collaborative thinking (Ruberg, Moore, & Taylor, 
1996). Scholars have reported student perceived learning to be correlated with the use of online discussion 
(Picciano, 1998, Jiang & Ting, 2000). Yet, lack of higher level learning (Bloom, 1956) in online discussion has been 
a drawback reported in many studies focusing on the effectiveness of this strategy (Garrison, Anderson, & Archer, 
2001, Gunawardena, Lowe, & Anderson, 1997; Schellens & Vackle, 2005; Sing & Khine, 2006). Given the 
complexity of today’s instructional and training tasks, the latest strategies used in face-to-face or online instruction 
should be designed to bring about all levels of learning. 

Participating meaningfully in an online discussion is a complex cognitive task. In the conventional online 
discussion format currently used in higher education, learners are provided with a discussion topic and some 
participation rubrics before they are immersed in the assigned discussion task. They are then expected to apply the 
participation rules, read and respond to other participants’ postings, figure out the interrelationships of materials 
being posted, deal with different styles in which the materials are presented, and finally cope with the technological 
problems that may arise during the discussion. Learners experience great amount of extraneous load within these 
activities, as well as germane and intrinsic cognitive load (Chandler & Seweller, 1991; Sweller, van Merrienboer, & 
Paas, 1998) presented by the instructional content of the discussion. Learners not only face new elements of 
information (e.g., the concepts, their theoretical framework, and identified information from other resources) that 
they need to learn, they also have to deal with the interactions among these new elements throughout the enumerated 
activities. Furthermore, as a result of encouraging participation in online discussion (Ruberg, Moore, & Taylor, 
1996), a large number of postings are created by students in the same class. Ideally, individuals should read each 
other’s posting, comment or reflect on them, and provide each other with responses. The cognitive resources needed 
in the process is far beyond the capacity of human working memory (Miller, 1956). 

In the present study we investigated the effectiveness of several cognitive load theory (CLT)-based 
strategies in facilitating higher level learning in online discussion. These strategies have been designed to reduce 
learners’ cognitive load in order to improve their learning performance. We applied these strategies in an online 
discussion and contrasted the outcomes with a conventional online discussion strategy. Following is a description of 
the online learning environment and the discussion tasks. 

Online Learning Environment 

An online course on Stress and Resilience in Families and Children was used in this experimental study. 

The course provided undergraduate majors in Family & Child Sciences with an introduction to family-based theories 
about the responses of children, adults, and families to stress. It examined ways of applying those theories to 
understanding the process of coping with stress and ways for professionals to intervene to help families in 
distress. The course objective was for the students to learn the techniques of working with families and children 
dealing with distress and supporting them in resolving the disruptions they were experiencing. 

The Family Distress Model (FDM) was grounded in research on social support and family problem solving 
(Cornille & Boroto, 1992). Families, based on this model, will cope with distress, depending on whether the stressor 
is a crisis or “just a problem”, following a predictable set of steps. The Family Outreach Model (FOM) was 
developed to identify specific strategies that persons can use to be supportive and effective when they form working 
relationships with families during times of distress (Cornille, Meyer, Mullis, Mullis, & Boroto 2008; Nalls, Mullis, 
Cornille, Mullis, & Jeter, 2009 ). This model grew out of the Family Distress Model and research about common 
factors of helping relationships (Hubble, Duncan, & Miller, 2002). In particular, the FOM links the stages of coping 
of families with the various styles and quality of helping relationships, suggesting the most effective style of helping 
relationship for each of the five stages, and strategies for using those specific styles. 

For this experiment, we selected the topic of Working Professionally with Families in Distress. This topic 
was on the course agenda for the week of 13 of a 16 week semester and focused on students understanding how 
persons who work with families in a professional capacity, such as teachers or health care providers, can develop 
effective helping relationships with children and their families. During this week, the participants learned how the 
Family Outreach Model and Family Distress Model could be used to help both the child and their family deal with 
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problems if the students selected career paths such as teaching, counseling, or social work. The discussion topic used 
for this week was designed based on the content and purposes of the discussion: 

When you think about your profession after you complete your education, what phase of the 
Family Distress Model best describes what the families you will encounter will be experiencing? What are 
the similarities between your expectations and others in this discussion? Using the framework provided by 
the Family Outreach Model, describe how you might work most effectively with families in that phase. In 
reading other participants ’ thoughts, share with one another how your expectations are similar and 
different from theirs? 

Students participated in group discussions that were organized around one of four strategies. 

Discussion Strategies 

Strategy 1 . Example-based postings 

The use of example posting aims at reducing the intrinsic cognitive load of the discussion task and the 
mental effort required for searching of appropriate steps to fulfill the task. In a complex learning situation, the 
learners need to understand abstract relationships of the concepts and how the abstract concepts fit with the schemas 
they already have. They need to process the relationships among the elements and be able to process multiple 
elements as one (Sweller & Chandler, 1994). Based on the findings of numerous research conducted on the 
effectiveness ofworked-out examples (i.e. Van Gog, 2006; Darabi, Nelson, and Palanki, 2007; Darabi, Sikorski, 
Nelson, and Palanki, 2006), we argue that an example-based posting that demonstrates what the learner is expected 
to do, reduces the intrinsic cognitive load involved in performing the task and frees the working memory capacity 
for higher cognitive activities, such as extracting new relations (analysis) and constructing new ideas (synthesis) 
(Bloom, 1956). 

Generally, an example posting helps students to understand the abstract concepts and relationships 
embedded in complex learning tasks by illustrating what those concepts and relationships mean in a specific 
situation. It also demonstrates the processes of how the learning task could be fulfilled, including how to analyze the 
elements of a situation, form new ideas, and how to make judgment base on certain criteria. Consequently, the 
learners’ attention is directed to the relations between relevant materials and relevant steps, which is similar to what 
a worked example (Sweller & Cooper, 1985, van Merrienboer & Sweller, 2005) would do. We contend that example 
postings reduce cognitive load by reducing irrelevant tasks, thus make the working memory available for activities 
fostering higher level learning. 

Strategy 2. Filtering messages 

Filtering message is a strategy that allows posting relevant messages together rather than scattering them 
around in a discussion board. Most of the discussion boards currently in use display messages under threads. It is 
very common that not all the messages contribute to the discussion threads. Often time a student will find only 
several of classmates’ postings make sense and he or she would not need all the information to form a good 
understanding of what is being discussed. In fact the non-relevant postings, we argue, may be distractive and thus 
causing extraneous mental load learners. By using filtering messages strategy, a user will be able to choose only the 
relevant messages to be displayed on his current screen. The messages irrelevant to current task will not be deleted 
and can be retrieved later for other tasks. By doing so, learner’s cognitive resource is concentrating on holding and 
processing relevant elements rather than irrelevant ones. 

Strategy 3. Limited number of postings on each page 

This strategy limits the number of postings on each page of discussion board to 7 ± 2 (Miller (1956). The 
popular online discussion boards nowadays (e.g., BlackBorad, WebCT Vista, and Moodle) display more than 20 
messages on each page by default. If all of the 20 messages provide new information, they will obviously exceed 
learner’s working-memory capacity and thus leave no extra cognitive capacity for processing abstract relationships 
of the 20 messages. Facing this cognitive overload, we argue, learners are not likely to achieve higher level learning. 
The design of limiting number of postings on each page of the discussion board is to break the new information into 
smaller chunks that hold no more than 7 ± 2 elements. So that students are able to preprocess the limited pieces of 
information before they move on to next chunk. 

Strategy 4. Combination Strategy 

This strategy combines the above three strategies of example posting, filtering messages, and limiting number of 
posting on each page. The messages are related to the current topic and include new ideas (no irrelevant messages) 
and there are no more than 7 ± 2 messages on each page. Using this strategy, the users should presumably be able to 
reduce the cognitive capacity needed for understanding the discussion task and save the cognitive resource for 
searching of appropriate problem solving processes. The users’ attention should be focused on relevant elements 
because irrelevant messages are filtered out and the number of message is limited to less than 9 including example 
posting. 
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Research Questions and Hypotheses 

Our research questions are: (1) Does each of the CLT -based online discussion strategies reduce cognitive 
load placed on learners compared to conventional discussion board? Which is the most effective? (2) Does each of 
the CLT -based online discussion strategies enhance higher level learning compared to conventional discussion 
board? Which is the most effective? (3) Does each of the CLT-based online discussion strategies enhance students’ 
performance on quiz? (4) Which discussion strategy is the most efficient in terms of mental effort invested and 
learning outcome? 

Assuming that cognitive overload is one reason for lack of higher level learning in online discussion, these 
strategies were designed to reduce cognitive load and therefore assist them with processing discussion content more 
efficiently for achieving higher level learning. Given the principles of CLT, We hypothesized that compared to the 
conventional method of online discussion, each of the proposed discussion strategies will result in: (1) a decrease in 
learners’ cognitive load; and (2) an increase in the amount of higher level learning. 

Method 

Participants 

Fifty nine undergraduate students enrolled in an online class in a large university in southeastern United 
States participated in the study. Six were eliminated from the study (reasons stated in the next session). The course 
offered an online discussion topic each week and students received credits for participation in online discussions. 

The majority of the participants were in majors related to family studies. Only two were from outside of the College 
of Human Sciences. All the participants were considered to have the similar pre -knowledge level of the subject. All 
of the participants were in at least the 3 rd year in their program. As to experience of online discussion tools, nine out 
of 53 had not taken any online class before. Considering that all the participants had used online discussion for the 
12 weeks before this study was conducted, we expected them to be equally experienced in using online discussion 
board. 

Procedures of the study 

The students were randomly assigned to one of five discussion groups: Example posting group, Filtered 
postings group, Limited number of postings group, Combination group, or Conventional group. All groups used the 
same discussion topic but different discussion strategies. Participants had access only to their assigned discussion 
group and were not able to see other groups’ discussion. 

Before the project began, it was approved by the Institutional Review Board for human subjects. Students 
gave their consent to participate in this study. They were asked to: (1) participate in the discussion at least once 
during week 13; (2) report their mental effort level right after they posted their postings for that week’s discussion; 
and (3) take a quiz at the end of the week. Six of the 59 students were excluded from the analysis because each of 
them had missed at least one of the tasks. Other than completion of a course assignment, no additional incentives 
were provided. 

Five groups 

Example postings group: This group used a discussion forum with example postings. Two postings were 
presented to the group together with the discussion topic. The first example posting presented a profession and 
described a family with which this profession might work. It analyzed the situation of the family using the FDM 
model and listed possible operations suggested by FOM model for working with the same family. The second 
posting, described another family that a different profession might encounter and referred to the characteristics of 
family described in posting 1 . It derives the abstract relationships between the conditions of the two families, and 
evaluated how posting 1 handled the situation. The participants in this group could construct their postings by 
referring to the scenarios provided in the two example postings and other participants’ postings. 

Filtered postings group: Participants in this group used a discussion strategy that only allows displaying 
information relevant to the discussion topic. The irrelevant postings were eliminated. Because none of the currently 
in use discussion boards has the function that allows the users to filter messages, we used a simulation for this 
group. We filtered and posted 10 messages in a simulated discussion board. Each actual participant saw 10 messages 
that seemed posted by 7 other students. Each participant saw the filtered content, but they could not see each other’s 
postings. This ensured that the information each participant saw was identical. 

Limited number of postings group: For this group we used a discussion forum that had limited number of 
postings on each page. In order to provide the same condition for participants in this group, a simulated discussion 
forum that contained pre-constructed postings by “fake” students were used. There were 14 discussion postings 
displayed in two pages (7 postings and the discussion topic on page 1 and 7 postings and a reminder messages on 
page 2). A note on Page 2 instructed participants how to go back to page 1 and how to link to the mental effort 
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measure. This simulated a discussion board that contained no more than 9 messages on each page. Participants were 
expected to read the messages on page 1 and process them before they move onto page 2. 

Combination group: For this group a combination of the three strategies (example posting + filtered 
message + limited number of message) was used. In addition to the discussion topic, 9 messages were displayed for 
these participants. The messages were carefully constructed by the researchers (including the instructor of the 
course) to discuss a new aspect of the discussion topic. Two of the 9 messages were example postings. 

Conventional group: As our control group, participants used a conventional discussion board strategy. They 
were presented with the discussion topic at the beginning of the week. Participants could see each other’s posting 
and were expected to read other’s messages to find the similarities and differences and develop their own reflection 
on the topic. 

Measures 

Higher Level Learning 

Content analysis of the discussion postings was used to measure levels of learning. Bloom’s (1956) six 
levels of learning were used as coding categories. They are knowledge, comprehension, application, analysis, 
synthesis, and evaluation, from low level to high level. To conduct content analysis, each posting was divided into 
meaning units so that each unit was coded into only one category. The primary author coded all the units according 
to the categories. A second coder was trained and asked to code a sample of postings (three postings) randomly 
selected from the data. There was 100% agreement between the coders. 

The following formula was used to calculate the percentage of higher level learning (PHL) for each 
posting. This percentage is the ratio of sum of the units coded as three higher levels of learning over the total 
number of units for all of six levels of learning. The higher the PHL value, the more percentage of higher level 
learning presented in a posting. 

N {analysis) + N {synthesis) + N {evaluation) 

PHL= 

N{aii) 

Mental effort level 

Participants were requested to report the level of mental effort they invested in composing a discussion 
message. Mental effort is measured using 9-scale instrument developed by Paas, Van Merrienboer, and Adam 
(1994). 9 is the highest level of mental effort, 1 is the lowest level of mental effort. 

Quiz score 

All the participants took an online quiz at the end of Week 13. The quiz was designed to test the students’ 
knowledge and understanding of the content discussed during this week. It consisted of 10 multiple choice 
questions. The participants’ score on this quiz was calculated as part of their final scores of the course. 

Instructional efficiency 

In order to identify which instructional condition is the most efficient in terms of less demand on mental 
effort and greater learning outcome, instructional efficiency (E) is calculated based on the formula suggested by 
Paas and van Merrienboer (1993). Mental effort level is standardized and denoted as Z(ME). Standardized score of 
higher level learning (Z(P)) was derived at by using the learners percentage of higher level learning as their 
performance scores and then instructional efficiency for each strategy was calculated using the following formula: 

„ Z(P)-Z(ME) 

£= Ti 



Results 

Strategies and mental effort 

A calculation of mean mental effort reported by participants in each group revealed that the participants 
using the conventional strategy reported the highest mental effort level (M=7. 14, S DM). 900). Compared to the 
conventional strategy group, the lowest mental effort exerted on the discussion of the topic was reported by the 
participants using the filtered posting strategy (M=5.75, SD=1.422). This was 19.5% lower mental effort than that of 
the conventional group. This was followed by the group using combination of the three strategies that reported 
13.45% lower mental effort (M=6.18, SDMJ.87). The group provided with limited number of postings on each page 
experienced medium level of mental effort (M=6.30, SD=1.337) and the participants in example posting strategy 
reported the least decrease in their mental effort (M=6.54, SD=1.05) 

Higher level learning 
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The results of content analysis of the discussion postings with regard to higher level learning shows that 
participants in control group using the conventional strategy of discussion posting indicated the smallest proportion 
of higher level learning (38.1%). Learners in limited number of posting strategy indicated the highest proportion of 
higher level learning (72.5%). Participants in example, filtered, and combination strategy groups displayed better 
percentages (64%, 58.0%, and 48.3%, respectively) of higher level learning compared to the participants using the 
conventional strategy. 

A One-way AN OVA test of group differences indicated significant difference between limited and 
conventional groups at .05 level (t=2.50, p=0.03). A significant difference was also found between example and 
conventional group at .10 level (t=1.78, p=0.10). According to the results of this examination, although participants 
in filtered and combination strategies outperformed conventional group, the differences were not significant. 

The analysis of learners’ performance on the quiz showed no difference across strategies. The quiz scores 
were found to be are positively skewed, with the range between 0 and 100 (100 is the highest possible score), mean 
of 74.3. The mean scores for participants in different strategies were as follows: Example group (M=65.4, SD=25.0), 
filtered group (M=77.5, SD=12.2), limited group (M=76.0, SD=12.7), combination group (M=77.3, SD=28.0), and 
finally the conventional group (M=74.3, SD=15.7). 

Instructional efficiency 

The analysis of the instructional efficiency data for each strategy, calculated according to the formula 
discussed earlier, indicated that the conventional discussion board was the least efficient (E=-.89). The filtered 
messages strategy was found to have the highest efficiency score (E=0.35) followed by the limited number of 
postings strategy (E=0.32). No instructional efficiency for example postings strategy was indicated (E=0.00). The 
combination strategy showed the lowest instructional efficiency (-.1094) among interventions right next to the 
conventional discussion strategy. A one-way ANOVA test of differences indicates that all of the four experimental 
conditions had significantly higher instructional efficiency comparing to conventional condition at .05 level. 

Discussion 

The results of this study clearly indicated that learners in conventional discussion group faced a heavy 
cognitive load placed by discussion task. Presumably the learners in online discussion processed part of the task and 
failed to perceive the latent relations and constructs of the materials which we contend was the reason for not 
achieving a higher level learning. Compared to the conventional group, participants in experimental groups reported 
lower level of mental effort indicating the success of our discussion strategies in reducing learners’ cognitive load. 
The example postings strategy provided users with connections between their existing schema produced by their 
prior knowledge and new abstract concepts and relations. Moreover, example postings, we contend demonstrated 
processes for performing the task which directed the learners’ attention to latent relations and relevant solutions. 
Similarly, filtered message strategy saved the cognitive resources needed to integrate relevant elements usually 
scattered around in discussion board. Limited number of posting strategy on each page broke the learning materials 
into several chunks of smaller size limited to working memory capacity to avoid overload. 

We expected that the strategy of combining the three experimental strategies would reduce cognitive load 
at least as much of the most successful individual discussion strategy. The results showed the combination condition 
reduced cognitive load by 13.45%, second to our more successful strategy, filtered message strategy (19.5%). Then 
the question became why the filtered messages strategy, when combined with other strategies, lost some of its 
impact in reducing the cognitive load while we expected more than when it was offered individually. One can argue 
that the fact that this successful strategy was combined with two others (example and filtered strategies) might have 
created a redundancy effect of strategies which lead to their combined lower effectiveness. They seemingly placed 
heavier cognitive load because (1) each of the two example postings, and each of the seven filtered messages 
contributed new information which participants might have perceived important for the task and (2) example 
postings were different from other messages in nature since they included instructor comments requiring more 
cognitive capacity. Compared example postings group, combination group had more elements to handle and 
compared to groups using filtered messages and limited number of messages, the combination group needed to 
handle messages different in nature. 

Another area of examination in this study was the impact of these strategies on the learners’ acquisition of 
higher level learning. Our content analysis results indicated higher scores for our discussion strategies in terms of 
this acquisition when compared with the conventional discussion method. Learners using the example postings and 
limited number of postings strategies demonstrated significant gains over their counterparts in conventional group. 
The filtered messages group had smaller gain. The group using the combination of the three strategies produced the 
least improvement in acquisition of higher level learning. 

Test scores have been regarded as measurements of performance. But no difference has been found in quiz 
score across groups in this study. Quiz score was not a good measurement in this situation because the online 
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multiple-choice quiz tested students’ lower level learning skills: recall of facts, understanding and ability to apply 
learned theory in given scenarios. Higher level learning was not the outcome for measure in the quiz, also higher 
level learning was not easy to be measured in multiple-choice quiz. 

We also examined the instructional efficiency for the experimental strategies used in this investigation. 
According to the results of our analysis, we found our experimental conditions to be significantly more efficient than 
conventional discussion board at .05 level. These findings indicated that the discussion boards embedded with each 
of the four strategies will enhance students’ learning with a lower level of mental effort. 

To sum up the findings of this study, we propose that the experimental strategies used in this study, 
compared to the more conventional method of online discussion, are more effective in terms of enhancing students’ 
gain in higher level learning. Because they were purposefully designed according to the CLT principles, they require 
less mental effort in part of learners using these strategies in their discussion of the online topics. Consequently, the 
strategies were found to be instructionally efficient in this learning environment. 

These findings have some implications for designing and developing instruction and training materials. 
Designers should make an effort to reduce learners’ cognitive load through the use of innovative strategies so that 
provide the learners with the opportunities for better performance. It seems that freeing some of the learner’s mental 
capacity from extraneous load, usually imposed by not desirable instruction, will result in higher achievements of 
learning. It also appears that as a result of this cognitive capacity that is freed by appropriate instructional strategy, 
the learners will have a chance for integrating, synthesizing, and evaluating the new information which lead to the 
higher level learning. 
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Abstract 

Recognizing perspectives of the members of university ensure success of ICT implementation. This study 
aims to compare and define some issues related to ICT use and expectations of students and instructors. 1282 
undergraduate students and 211 instructors were participated in the study. Results showed that instructors have low 
expectancy while students have a higher expectancy related to bringing of ICT in education. Students want both in- 
class and online technologies while instructors believe the crucial need of in-class technology use. According to the 
instructors, technology use is less because of the technical deficiencies and lack of time, while the students argue 
that lack of knowledge of instructors cause less technology use in education. 

Introduction 

Today most of the universities develop some strategy plans and prepare investments for technology 
integration. On the other hand, almost all of them have some problems to convince all university members to use 
these technologies. According to Straub, Keil and Brenner (1997), ICT use is not only shaped by technical situations 
but also it is shaped by the nature of social environment, organizational dynamics and economical and psychological 
issues. Social pressure is one of the crucial issues while developing an acceptance model because this is a good 
motivation to encourage to use innovations (Anandarajan, Igbaria & Anakwe, 2002). Thus, in this study looking 
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from two different perspectives, instructors and students, seemed important to understand whether both instructors 
and students have same language and whether students’ expectations shape the instructors’ use of technology. 
Researchers also proposed to look both students’ and instructors’ perspective to understand how they interact with 
each other in terms of technology use. Especially, this survey study is important to reveal the students’ expectations 
from instructors and instructors’ views on these expectations. 

Higher education is significantly affected by the information and communication technologies. Technology 
use and integration is precious and return of this investment turns back in the long run. Therefore, this decision 
needs to be made with a careful planning (Mumtaz, 2000). Planning should begin with the awareness and readiness 
of users of technology and the view of all stakeholders should be considered. Instructors’ and students’ attitudes and 
motivations toward these technologies and their usage play a critical role to diffuse these technologies into 
educational settings. Understanding their attitudes and the reasons of avoidance provide us to present solutions. For 
this reason, opinions and expectations of these active users who use these technologies is very important. Effective 
technology integration can be provided if the technology is indeed available, if people are aware of it and if people 
are motivated (Perkins, 1985). The purpose of the study was to investigate the current situation of ICT use, opinions 
and future expectations in terms of integrating ICT in higher education from the students’ and instructors point of 
view. 

This study will be helpful for people who work on technology integration in higher education. In higher 
education it is not enough to take the opinions and expectations of faculties. Students’ opinions are also crucial, 
because they are the ones who are directly affected by the use of technologies. They might also provide a technology 
integration suggestions if they really aware of benefits of technology. Looking from two perspectives will point out 
that total awareness of technology use in education. This study might also provide some findings related to conflict 
between instructors and students in technology use about the ways of effective technology use for to motivate 
students and to increase the effectiveness of instruction. Therefore, this study might provide new insights to see 
conflicts in technology integration and insights related how technology integration should be planned by considering 
each component of the system. 

Method 

The sample of the study consisted of 1282 students (639 males and 643 females) enrolling at Middle East 
Technical University (METU) and 211 instructors (119 females and 92 males) working at METU. Students were at 
least 1st grade attending a common course from the Modern Languages Department. Ratio of the students from the 
Faculty of Engineering was 48.1%, 18.3% from Faculty of Art and Sciences, 14.4% from Faculty of Economic and 
Administrative Sciences, 11.8% from Faculty of Education, 5.2% from Faculty of Architecture and 0.7% from 
graduate programs. The distribution of students’ faculties represents the actual ratios of the students’ profile at 
METU. Most of the students (84.5%) had been at METU for at least three years. This showed that students were 
knowledgeable at least about using Internet and computer technologies which is used generally at METU. 

Within 211 faculties, 52 of them were professors, 38 were associate professors, 3 1 were assistant professors, 15 
were lecturers and 69 were research assistants. The faculties’ working period at METU ranged from 5 to 32 years. 
They have been using computers for 28 years on average. Their computer use per week was 30 hours on average, 
while their internet use was 1 1 hours per week. 

Data were collected with two questionnaires which were different for students and for instructors. This 
questionnaire consists of close-ended and open-ended questions. It is divided into three sections. Students’ 
questionnaire consists of 5 demographic questions. Second section includes 7 questions about technology use in 
education from students’ points of view. These questions also have open-ended part to write their opinions. Last 
section covers 2 open-ended questions about the deficiencies in instructional technology use from students’ points of 
view and their suggestions for modern campus. 

Instructors’ questionnaire consists of three main parts. First part includes gender, title, age and department, 
the time that instructors have been working at METU, weekly computer and Internet use related 6 demographic 
questions. The second part consists of 13 questions focusing on the current situation of the use of instructional 
technologies in courses, including questions targeting on whether the instructors are using web-based instructional 
tools, the frequencies of instructors’ using instructional technologies in the courses, whether the instructors are 
thinking that they are using instructional technologies effectively, if there any technologies that they would like to 
use in their courses but could not use, whether Internet and computer use have a positive impact on the education 
given at METU, whether they are aware of and using Smart Classes at METU. The third and the last part of the 
questionnaire consisting of three open-ended questions, focuses on the future expectations of the instructors, the 
encouragements, support from the university for integrating instructional technology into education, and the 
technologies they would like to use in their courses in the future. 
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Results 

Results of this study was presented in the parts of current situation for technology use, deficiencies in both 
perspectives, expectations from technology use and expectations from future. Students and instructors differ in terms 
of expectations and students have more positive views and expectations from technology. However it was observed 
that students do not like totally online classes. Both instructors and students mostly preferred to in-class technologies 
to comprehend content easily, however very few students’ stated that technology makes courses attractive. 
Instructors also agree with students about technology improve quality of instruction. 40.2% of students expected 
regular handouts, 16.4% of them expected PowerPoint presentations with projectors, 56.6% of them stated they 
expected a rich learning environment with simulations and animations, and 45.4% of them wanted video related the 
topic. Some students stated that they hate presentations which instructors readjust like a book. They also expected 
specialist people related to topics, active participation, discussion environment and more practice. To provide active 
learning, 45.4% of students expected Internet based communication tools which provide effective communication 
with instructor, and most of students (75.5%) stated that they wanted a course web site they easily find lecture notes 
and additional resources. LMS (NETclass) used by METU also demanded by 21.8% of students, because it can 
provide most of web applications students demanded. However, students expected full online courses rather less 
than students wanted other kind of educational opportunities, 10.7% of students stated they wanted online courses. 
Most of instructors stated that they use technology effectively by 54.9%. 

84% of them answered the open ended question; “why technology use is important for your department?”. 
About 24% of them stated that visual materials and tools provide easy presentation, 20% of them stated that their 
course requires technology and 13% of them thought that it provides time management. “Other” parts, of open 
ended questions revealed that if the instructors can follow developments in the world, courses become more 
attractive, students learn effectively and interaction increases by means of technologies. On the other hand, the 
students argued that instructors tend to use traditional methods. According to the students, most of the instructors 
use blackboard in classroom by 78.6%, and namely hard copy by 52.3%, overhead-projector 65.6%, computer- 
internet by 29.3% and a few video by 6.6%. Students accepted that instructors use computers in classroom but they 
also stated that use of computers and projectors is limited to presenting the text based content. This situation shows 
that although there is a rapid change in technologies, traditional habits in instruction is difficult to change. On the 
other hand, most of instructors argued that they use technology effectively; however some of them reported that they 
sometimes have problems with hardware, lack of support personnel, and lack of training about education and 
technology integration into their course content. 

Most of the students (90.5%) use the Internet (90. 5%), computers ( 79.8%), hard copy material (72%), 
communication tools like chat-forum- instant messaging (34.2%), cell-phone (19.5%) and sound-recorder (4.7%) for 
educational purposes. Most of them thought that online courses are limited with online resources and assignments in 
the university learning management system. Students who stressed the inadequate knowledge of instructors about 
technology integration commented on the open-ended part of the deficiency question. In addition to faculties’ lack 
of knowledge, students stated that faculties do not want to use technologies even if required conditions are available. 
Other deficiencies related with technology use by faculties are that faculties find technology use difficult, tell the 
lecture as reading texts from presentation, use technology ineffectively, do not spend time for technology 
integration, and they are unwillingness to use technology. 

Both students and instructors were asked what they would like to see as technology in classrooms with 
open ended questions. First five preferences of students are access of online course content (13.8%), access of video 
and sound recording of courses (6.8%), online elective courses (5.7%), improvement of technological infrastructure 
(4.8%) and more course-related technologies like physical experiment settings (4.3%). On the other hand, 
instructors’ choices are a bit different from students. 58 of instructors answered this question and ratios were 
calculated in accordance with number of 58. Instructors stated that if they had a chance, they would like to use smart 
boards in their classroom by 27.5%. Smartboards are becoming common at METU, however most of them do not 
use them. Other preferences of instructors are namely computers (31.0%), projectors (24.1%), Internet technologies 
(13.8%) and software like modeling or simulating (12.1%). Considering the departments of the students and 
instructors, it is obvious that engineering departments need to use simulation and modeling. These results implied 
that instructors are in need of in-class technologies. On the other hand, students would like to reach course content 
anytime everywhere, thus they want mobilized course content mostly. 

Most of instructors (21.4%) stated that technology use is limited because of the insufficient hardware. 
12.7% of them also stated that the time limit in their classes is the reason for insufficient technology use. Other 
reasons they stressed are crowded classrooms (11.8%), lack of knowledge about technology use, lack of knowledge 
of integration with technology and course content, and insufficiency of software (9.1%). Open ended-questions also 
supported these findings, instructors stated that they have problem with inaccessibility of hardware, lack of support 
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personnel and lack of training about education and technology integration. According to the findings from student 
questionnaires, open ended-questions also revealed similar results. They stressed the biggest deficiencies in 
technology use is insufficient computer related hardware (10.5%), inadequate laboratory conditions (7.6%), lack of 
Internet tools use (7.4%) and lack of instructors knowledge about technology (7.2%). 

Conclusions 

In developed countries interest in use of technology in classroom has shifted to effective use of these 
technologies. However in Turkey, we still need more technology integration. Especially students have a big 
prejudice about online courses (Inal, Karakus & Cagiltay, 2007). The study showed that students want to have 
online access to course content, while they are not willing to take online courses. Therefore, new studies should 
investigate how students’ and instructors’ perspectives can change in favor of online education and underlying 
reasons of this hesitation. They want to be active in classrooms and to develop a warm relationship with instructors 
by means of online communication tools. Classroom and laboratory conditions and technology fluency of instructors 
are seen two major obstacles for technology integration. Results suggested that instructors should be trained about 
how to use the Internet and multimedia technologies effectively in their courses. Using technology should not be 
meant as teacher-centered instruction but as students suggested, technologies should be used to make students 
active. Communication tools should be provided by university to provide student-instructor interaction, thus this 
active participation can be achieved. 

Students expect quick feedback, timely answers about course context, and effective guidance of facilitator, 
broad content, more social interaction and communication opportunities for online education (Inal, Karakus & 
Cagiltay, 2007). These expectations also consistent with Langenberg and Spicer (2001)’s investigation that 
instructors are turning to “knowledgeable mentor ” and “learning facilitator ” in education. Lack of meeting these 
expectations might cause the students’ disagreement with online education opportunities providing a high quality for 
university education like in this study. Therefore, universities should improve the quality of online courses by 
considering students’ expectations. This study’s results also improve this finding by its results that students prefer 
online opportunities to communicate with instructors and to reach course materials easily more than online course 
opportunities. 

This study presented a current situation of technology use in perspective of students and instructors in the 
one of the most leading university in Turkey. Study results showed that both students and instructors are willing to 
use in-class technologies related to subject matter while they hesitate to use online technologies totally. METU, as a 
modern university, can minimize the deficiencies, pitfalls and problems, and support the expectations with a 
carefully strategy plans. This study might be helpful for planning these strategies by presenting the current situation 
of technology use from instructor’s perspectives. 
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Abstract 

As more students become comfortable with sending and receiving text messages, text messaging is 
emerging as a tool educators are using for multiple purposes in both face-to-face and online courses. This research 
study investigated the impact of using text messages to convey course information for six online courses, taught by 
one instructor, as measured by the Community of Inquiry survey instrument. Text messages communicated course 
information, including reminders of assignment due dates, notification that assignments have been graded, and short 
feedback on discussion postings. Students completed a Community of Inquiry (COI) survey and a survey on their 
reaction to using text messages in the course. Overall results indicate that students reacted positively to receiving 
text messages. COI survey results indicated, however, that there were no significant differences between students 
who received text messages and students who did not receive text messages in their perception of social presence, 
cognitive presence, and teaching presence as represented by the COI framework. This lack of significance may be 
partially attributable to the instructor’s extensive prior experience and success in online teaching. 

Introduction 

The current trend in research regarding cell phone usage in higher education appears to privilege its use as 
a learning tool or instructional delivery device. In the research, two terms associated with cell phone use are used 
interchangeably, namely, text messaging and Short Messaging Service (SMS). Both of these terms refer to the 
ability to send and receive short text messages on a cellular device. A third term, instant messaging (IM), is also 
found in this body of literature. Instant messaging refers to conversational and immediate text communication 
between two people. IM communication can occur between two computers, two phones, or a computer and a phone. 

Mobile technologies, especially cellular phones, are intriguing to educators because these technologies are 
small, relatively inexpensive, and ubiquitous — almost every student has a cell phone. As Markett, Sanchez, Weber, 
and Tangney (2005) note, mobile phones “require no technology training, and are not intimidating” (p. 282), 
therefore their use is often seen as low risk and low cost. However, the use of cell phones in education is often 
contentious. Some educators support the idea of exploring educational uses of cell phones (Kolb, 2007), while other 
educators stress the potential problems that cell phones can introduce in educational settings (Allen, 2007). One 
often mentioned potential problem with cell phones is the ability of students to text each other information related to 
tests. For example, in a nationwide survey (Obringer & Coffey, 2007), 80% of the high school principals that 
completed the survey indicated that they felt text messaging is a problem, or a potential problem, during tests and 
examinations. Those who argue in favor of using cell phones point to the potential positive aspects of text 
messaging, including the ability to help students learn course content, help people change their behavior, and help 
build a stronger community of learners in online courses. 

In the area of learning course content, learning vocabulary is a common focus of text messaging research. 
Researchers such as Lu (2008) examined the effectiveness of Short Messaging Service (SMS) on second language 
acquisitions, while Cavus and Ibrahim (2009) used SMS to support the learning of new technical English language 
words. In another study, Thonton and Houser (2005) found Japanese students learned significantly more vocabulary 
when they received vocabulary lessons via text messages on their phones compared to students who received the 
same information in a paper format. Goh and Hooper (2007) explored the use of SMS crossword puzzles as a means 
to engage students and to promote interaction through learning activities, and Uzunboylu, Cavus and Ercag (2009) 
used SMS combined with student photographs to portray local environmental blights thereby increasing students’ 
environmental awareness. Text messaging has also been studied in the context of changing behavior patterns. 

In a study by Riley, Obermayer, and Jean-Mary (2008), student smokers were aided in a smoking cessation 
intervention through the use of a combination of text messages and web-based resources. The outcome indicated 
positive results in the number of study participants who quit smoking and those participants who continued to smoke 
reported reduced smoking rates. Another example of the use of text messaging in relation to health behaviors is a 
study by Gerber, Stolley, Thompson, Sharp, and Fitzgibbon (2009). In this study, healthy eating and activity tips. 
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expressions of encouragement, and general health text messages were sent three times a week to participants who 
had already undergone a weight reduction program. These messages were designed to encourage continued healthy 
behaviors. In a review of fourteen studies of text messages used to support changes in health behavior, Fjeldsoe, 
Marshall, and Miller (2009) found thirteen of those studies showed positive behavior changes, especially when 
messages were tailored to a specific situation. 

While these aforementioned studies are all very worthwhile endeavors, there is a gap in the research 
regarding the use of cell phone text messaging as a tool for learning course content or affecting behavior and using 
text messaging for building a community of inquiry and promoting interaction in online courses. 

Although there are many opportunities for students and the instructor to interact in an online environment, text 
messaging provides a means of contacting students about the course in a way that does not necessitate them being 
online and takes advantage of the ubiquity of cell phones on today’s college campuses. Additionally, research has 
pointed to the importance of creating a Community of Inquiry (COI) in the online environment (Anderson, 2004; 
Garrison, Anderson, & Archer, 2000; Garrison & Arbaugh, 2007; Swan, 2001). A COI as defined by Garrison et al. 
(2000) refers to the environment created by teachers and students where meaningful learning is defined by the elements 
of cognitive presence, social presence, and teaching presence. 

Cognitive presence (Garrison et al., 2000) is defined as the extent to which participants are able to construct 
meaning through sustained communication. Categories for cognitive presence include triggering events, exploration, 
integration, and resolution. Shea, Pickett, and Pelz, (2003) elaborate the term by adding that it is achieved in concert 
with satisfactory social presence and effective teaching presence. Examples of cognitive presence include student 
questions, expressions of confusion and indictors of students making connections and building new knowledge. 

Kehrwald (2008) defines social presence as “... the means by which online participants inhabit virtual 
spaces and indicate not only their presence in the online environment but also their availability and willingness to 
engage in the communicative exchanges which constitute learning activity in these environments” (p. 94). Garrison 
et al. (2000) describe social presence as the ability to project oneself as a real person (one’s full personality) both 
socially and emotionally in the online environment. The original categories of social presence, as defined by 
Garrison and Arbaugh (2007) are affective expression, open communication, and group cohesion. Social presence 
can be identified by the use of emoticons to express oneself, idioms and more informal and familiar language, and 
evidence of students helping each other out in a collaborative manner. Garrison and Arbaugh also suggest that 
activities which cultivate social presence can enhance the learner's satisfaction with the Internet as an educational 
delivery medium and to that we add that the use of SMS also supports student satisfaction. 

Teaching presence as defined by Anderson. Rourke, Garrison, and Archer (2001) is “. . .the design, 
facilitation, and direction of cognitive and social processes for the purpose of realizing personally meaningful and 
educationally worthwhile learning outcomes” (p. 5). Based on this definition, the authors distill three components or 
elements of teaching presence: instructional design and organization, facilitating discourse, and direct instruction. 
Examples of teaching presence are the instructor resolving misunderstandings and correcting misinformation that 
appears in discussion postings, moving the discussion along in a timely manner, keeping the discussions relevant 
and on track, as well as encouraging students to participate online and share their thoughts and ideas. 

Due to the widespread use and support of the COI framework for defining attributes of effective online 
learning (Garrison & Arbaugh, 2007), the COI instrument was selected as the tool to measure student perception of 
teaching presence, social presence, and cognitive presence in this study. 

Research questions 

This research project investigated the impact of text messages on student perception of the sense of 
community of inquiry and student reaction to receiving cell phone text messages conveying course information and in 
online courses. The specific research questions were: 

• Does the use of text messaging in online courses increase student perception of the community of inquiry? 

• What is student reaction to receiving text messages related to course content? 

Methodology 

Participants in this study were students in six graduate-level courses, taught online by the first author, at a 
large Midwestern university during two consecutive academic semesters. Students were solicited to participate in the 
study during face-to-face orientation sessions and also via email. Since the study entails the use of a cellular phone 
and text messages, some students were unable or unwilling to participate due to a number of reasons, including not 
owning a cellular phone, not having unlimited text messaging, or simply not wanting to receive text messages. All 
students who volunteered for the study signed a consent form. Students in the study were randomly assigned to one 
of two groups. One group received text messages throughout the semester; the other group did not receive text 
messages. To avoid inundating students with text messages, the consent form indicated that students would receive 
no more than two text messages per week. A total of 52 students signed up for the study. 
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Students in the study supplied a cellular phone number to the instructor and indicated the types of messages 
they wanted to receive. There were five options for types of messages students could receive: 1) reminders of due 
dates for assignments; 2) important vocabulary; 3) short feedback on discussion postings; 4) notification that an 
assignment had been graded; and 5) course related teasers (e.g.. quality web sites related to course content, links to 
online videos related to course content, relevant research articles). All students elected to receive text messages to 
remind them of due dates for assignments and to be notified when an assignment had been graded. The other options 
were chosen less frequently. Approximately half of the students elected to receive all five types of messages. There 
was no limit regarding how many of the five types of messages the students could choose to receive. 

Procedures 

At the beginning of each semester, the first author entered student-supplied cellular phone numbers into her 
cellular phone, along with the student’s name. During the first week of class, the instructor sent a text message to 
each student to insure that text messages were received. Throughout each semester, a one or more of the five types 
of text messages were sent to students. The most frequently sent messages had to do with due date reminders and 
notification that assignments had been graded. 

The instructor typed and sent all text messages through her cell phone rather than using an online text 
messaging service. While this approach added time and complexity to the process, the instructor felt it was 
important to keep student cellular phone numbers private and to control when text messages were sent. 

Due date reminders were sent approximately two days before assignments were due, but were only sent to 
students who had not yet submitted the assignment, therefore even if a student was participating in the study, that 
did not mean the student would receive a particular text message. A typical reminder was, “Just a reminder that your 
project 3 is due by Wednesday, Feb 25 by 1 1:00 pm.” 

In order to take advantage of the immediate nature of text messages, text messages to alert students that 
projects had been graded were sent within minutes of a grade being posted in the online grade book. A typical text 
message alerting a student that a project had been graded was, “I’ve graded your tool project. Excellent! You earned 
the max # of points.” Since the instructor wanted students to access the course management system to review 
detailed feedback, the text message only indicated that the project had been graded and the number of points earned. 

An example of a discussion posting text message was, “I enjoyed reading your posting about mobile 
learning. I can see from your post that you have given the use of mobile technologies serious thought and 
consideration. It will be interesting to learn about the results of your trial of using text messaging with your 
students.” For vocabulary, one of the terms defined through a text message was cloud computing. The text message 
to define cloud computing was lengthy and time-consuming to enter. An example of a teaser was a text message 
about a new technology, siftables. The text message of the teaser was, “Check out this video on siftables. Looks 
pretty cool, http://siftables.com” At the end of the semester, all students who participated in the study were asked to 
complete the validated Community of Inquiry survey instrument ( Arbaugh, Cleveland-lnnes, Diaz, Garrison, Ice, 
Richardson, & Swan, 2008; Shea, Swan, Li, & Pickett, 2005) and those who received text messages were asked to 
complete the Cell Phone Text Messaging Survey. 

The COI survey consists of 37 Likert scale items with a score of 1 representing strongly disagree, 2 
disagree, 3 neutral, 4 agree and 5 representing strongly agree. Additional multiple choice questions included were: 

1) What grade do you expect to receive in this course? 2) Gender 3) Age Group 4) Did you receive text messages 
during this course? 5 ) How many online course have you taken? 

The Cell Phone Text Messaging Survey is designed to measure student reactions to and impressions of the 
cell phone text message strategy. This survey consisted of 13 Likert scale questions with two open ended questions 
soliciting additional comments about receiving text messages and ways to improve text message usage within the 
course. 

Results 

A one-way ANOVA was computed to compare the Community of Inquiry survey score across the two 
levels of cell phone text messaging. The dependent variable was the score from the COI survey and the independent 
variable was the cell phone text message, received yes or no. 
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Table 1 

ANOVA Results of Community of Inquiry Score by cell phone text message 

ANOVA 



COI 





Sum of Squares 


df 


Mean Square 


F 


Sig. 


Between Groups 


560.784 


1 


560.784 


2.623 


.114 


Within Groups 


7911.882 


37 


213.835 






Total 


8472.667 


38 









There was not a significant effect of cell phone text messaging on the COI score at the p <.05 level for the 
conditions ( 1 ,37) = 2.623, p = .114, n = 39. R Squared = .066 (Adjusted R Squared = .041), therefore only 6.6% 
of the variability in the COI score is explained by the cell phone text messages. 

Cell Phone Text Messaging Survey 

The Cell Phone Text Messaging Survey is based on similar surveys found in the literature. There were 
fifteen items in the survey instrument, thirteen Likert-scale items and two open-ended items. Only students who had 
received text messages completed the survey. A total of 24 students completed the survey. Although students were 
reminded at least twice to complete the survey, not all students who received text messages completed it. Table 2 
contains results in percentages. 

Descriptive statistics indicate that the majority of students agreed or strongly agreed that text messages 
were a good way to receive course information (95.2%), that receiving text messages were useful (91.3%), and that 
receiving text messages was fun (95.2%). Over eighty percent of the students (82.6%) indicated that it is normal for 
them to receive text messages, and even though the majority of students did not think that receiving text messages 
was annoying, there were students (13%) who thought receiving text messages was annoying at times. A majority of 
students did log into the course management system shortly after receiving a text message (82.6%) and they were 
either neutral about (30.4%) or liked (69.6%) being able to determine the types of messages they received. Similar 
results were obtained when students were asked about their satisfaction with the number of text messages they 
received. Thirty percent were neutral about the number of messages received, while 69.6% agreed or strongly agreed 
that the number of messages received was about right. 

Over half of the students indicated they did not text back (52.5%) to the instructor and only 13% of the 
results indicated that a text message to the instructor was initiated at least once during the semester. Less than 10% 
of students thought that the text messages did not help them stay up-to-date with the course, a little over one-quarter 
of students were neutral on this item (26.1%), but the majority (65.2%) either agreed or strongly agreed that 
receiving text messages did help them stay up-to-date. Just over ninety percent of the students (91.3%) felt that text 
messaging can serve a useful purpose in education and over eighty percent (82.6%) felt that receiving text messages 
was a positive aspect of the course. 



124 





Table 2 

Cell Phone Text Messaging Results 



Survey items 




Strongly 

disagree 


Disagree 


Neutral 


Agree 


Strongly 

agree 


Receiving text messages on my cell phone is a 
good way to communicate course information. 


0.0% 


0.0% 


4.3% 


34.8% 


60.9% 


I received course information that was useful 
to me through text messages on my phone. 


4.3% 


4.3% 


0.0% 


34.8% 


56.5% 


I tended to login to Vista within several hours 
of receiving a text message. 


8.7% 


8.7% 


26.1% 


26.1% 


56.5% 


Receiving text messages was a fun way to get 
course information. 


0.0% 


0.0% 


4.3% 


34.8% 


60.9% 


Receiving text messages was annoying at 
times. 


60.9% 


13.0% 


13.0% 


8.7% 


4.3% 


I frequently texted back to the instructor after 
receiving a text message from her. 


34.8% 


17.4% 


21.7% 


8.7% 


17.4% 


I constantly use my cell phone, so receiving 
text messages is normal for me. 


4.3% 


4.3% 


8.7% 


26.1% 


56.5% 


I liked being able to determine the types of text 
messages I would receive. 


0.0% 


0.0% 


30.4% 


34.8% 


34.8% 


I received just about the right number of text 
messages from my instructor per week. 


0.0% 


0.0% 


30.4% 


34.8% 


34.8% 


I initiated text messages to my instructor at 
least once during the semester. 


52.5% 


21.7% 


13.0% 


8.7% 


4.3% 


Receiving text messages helped me stay up-to- 
date with this course. 


0.0% 


8.7% 


26.1% 


17.4% 


47.8% 


Text messages may serve a useful purpose in 
an educational setting 


0.0% 


4.3% 


4.3% 


30.4% 


60.9% 


Receiving text messages was a positive aspect 
of the course 


0.0% 


0.0% 


17.4% 


21.7% 


60.9% 
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Students who provided responses to the open-ended questions were positive about receiving text messages. 
Examples of statements include: 

“The text messages about links to different technology (siftables) was really engaging. It 
helped [me] to stay engaged in what we were working on. ” 

“I loved receiving text messages with course info and grades! It was never annoying. ” 

“It was very rewarding to receive a positive text message like, ‘Good job on your 
journal. ’ ” 

“The text messaging kept me on track and I felt connected to the teacher for the course. The 
feedback was very beneficial and I’m so thankful that I was able to participate. This was my 
first experience with online course work and the texting helped me feel in touch. [The 
instructor] does an awesome job with keeping online learners actively engaged!” 

Although few suggestions were offered for how to improve the use of text messages in the online classes, 
one student recommended that the first message be an introductory message that clearly identifies that the message 
is from the instructor. Another student recommended that “other professors should use this technique to 
communicate.” 

Discussion 

This study was undertaken to investigate the effect of cell phone text messages on student perception of the 
effectiveness of the learning environment they experienced in online courses, as measured by the Community of 
Inquiry framework. The researchers sought information to inform how the use of text messages in online courses 
could enhance the online learning experience. While the results of the COI survey did not show significant 
differences between students who received text messages and students who did not receive text messages, there 
were, nevertheless, important and noteworthy results. 

First, the instructor found that texting students contributed to a stronger feeling of connectedness to those 
students. In much the same way that students appreciate interaction from an instructor, many instructors are pleased 
when students engage in both formal and informal modes of communication with them. Even though only a few 
students sent text messages back to the instructor, those who did opened up additional lines of communication and 
reinforced the use of text messages and the notion that text messages can be a positive aspect of an online learning 
experience. 

Second, the use of text messaging appealed to encourage students to log into the course management 
system (over 80% of students indicated they logged in shortly after receiving a text message). Since much of the 
work of online courses takes place within a course management system (e.g., online discussions, emails, course 
announcements, submission of assignments), when students log in on a frequent and regular basis, they may be more 
successful in meeting course goals, feel more connected, and be less prone to drop out since they are more apt to be 
up-to-date on course expectations, requirements, and announcements. 

Third, students were positive about receiving text messages. As results from the Cell Phone Text 
Messaging Survey indicate, students thought that communicating course information via text messages was a good 
idea as well as a useful one. Giving students choice and control over the types of messages they received was also 
seen as positive by students. 

Several aspects of this research proved challenging and should be considered in similar studies of this 
nature. When the study first begins, the instructor should clearly indicate whether he or she is open to receiving text 
messages from students. Students were unsure as to whether they were to text back to the instructor or initiate text 
messages to the instructor. Informing students that they were welcome to send text messages to the instructor may 
have further enhanced communication about course content as well as increasing the students' perceptions of social 
presence. As one student noted, “I guess I didn't realize that texting back was an option. If I had been more aware of 
that I would have definitely sent back information.” 

The process of composing and sending text messages is another area warranting careful thought and 
planning to better control the level of complexity in sending text messages. The instructor compiled a list of students 
participating in the study along with the types of text messages these students elected to receive. In order to send 
reminders of due dates, for example, the instructor had to refer to the list to determine if a particular student was to 
receive a text message, verify which type of text message they were to receive, insure that the student had not 
already turned in the assignment to be graded (early), compose the text message, and finally send it. The instructor 
also discovered that providing feedback on discussion postings via cell phone text messages was extremely time 
consuming, since to be most effective, the instructor felt the text message needed to be customized to a student’s 
specific posting. In addition, these text messages were fairly lengthy. For these reasons, only a few text messages 
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were sent providing feedback on discussion postings. An option for future consideration is the use of pre-written 
posting responses that would be, of necessity, more general in nature, but could nevertheless convey the instructor’s 
assessment of the overall quality of the posting. 

Two types of the five text messages provided were problematic; text messages for vocabulary and text 
messages for course content teasers. Since vocabulary is addressed in this instructor’s courses through targeted 
terminology assignments, there were not many additional vocabulary terms that students needed to learn. And, with 
the high level of activity in online courses, the instructor struggled with what types of teasers to send, as well as 
when these types of messages are best sent. 

In addition to the above challenges, there are several limitations in this study worth noting. First, this study 
was conducted across several online classes taught by the same instructor. Participants in this study were very 
satisfied with the instructor, rating her 4.8 on a scale of 5.0. Secondly, students rated their overall satisfaction with 
the course as 4.66, and they rated their learning 4.56 out of 5.0. Because of their satisfaction with the course, their 
learning, and the instructor, the findings of no significance in this study appear logical. The instructor has many 
years of experience in teaching online and is well versed in effective online teaching strategies. Because of these 
high average scores regarding student satisfaction, it is possible that the addition of cell phone text messaging did 
not significantly add to students' perception of a community of inquiry because teaching presence, social presence, 
and cognitive presence were already established. The authors encourage further investigation into the effect of cell 
phone text messaging on student perceptions of a community of inquiry across a variety of instructors and with a 
larger sample size. 

Convenience sampling was used, from which the cell phone participants were randomly assigned to 
participate in the study; however non-probability sampling limits the generalizability of the results to other similar 
populations, and introduces sampling bias into the study. It is also important to note that participants in the study 
were limited to those who had cell phones and who were willing to incur possible additional fees associated with 
text messaging. 

Conclusion 

As instructors continue to explore ways to enhance online learning experiences, text messaging may be one 
technique worth further investigation. In this study, mean scores on the COI instrument indicated that student 
perception of the components that comprise the Community of Inquiry framework were high. Students were also 
highly positive toward the use of text messages, as reflected in the Cell Phone Text Messaging Survey. Student 
comments further support the positive reaction to the use of text messaging to convey course information. As one 
student stated, “I would just continue with it.” 

Future studies may want to limit the types of text messages students can receive to two or three, rather than 
five; include larger sample sizes and multiple instructors; and investigate the use of online text messaging services. 
Additionally, other frameworks for investigating the educational and social effects of instructor initiated, course 
relevant, cell phone text messaging should be considered. 
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Online, Face-to-Face, and Flybrid Learning: 

Which Model Takes the Most Time? 

Dianne Ford Lawton 
Troy University — Brunswick 
104 Lagoon Way 
Brunswick, G A 31525 

Abstract 

This article compared the actual amount of time that graduate students enrolled in three different learning delivery 
models spent completing assignments. It was hypothesized that graduate students enrolled in online classes would 
spend more time in class preparation than students enrolled in either face-to-face or hybrid classes. Participants in 
the study consisted of 10 online students, 10 face-to-face classroom students, and 27 hybrid classroom students. 
While the course content was the same for all participants, the difference among the three groups was the delivery 
model. Data gathered through surveys were analyzed by ANOVA in order to measure whether or not there were 
significant differences between the group means. The online students reported that significantly more time was 
required to complete their work than did students in the hybrid group. 

Technology is changing the way that teachers teach and students learn. The Internet allows students to 
connect, collaborate, and engage the world (Tapscott, 2009). Garrison and Vaughan (2008) point out that 
universities and colleges must keep up with these changes in order to make learning relevant and engaging for 
today’s student. Another challenge facing colleges and universities is the shift in demographics. According to 
Ausburn (2004), part-time adult learners are the new majority in higher education, comprising over 50% of the 
postsecondary student population. Robinson and Hullinger (2008) posit that the advent of the Internet and the Web 
has brought sweeping transformation to higher education and has facilitated learning environments for online 
classes. Ausburn also notes that nontraditional working adults, age 26 and older, comprise the largest group for 
online learning. These students often work full-time and cannot commit to traditional classroom based instruction 
that follows a rigid schedule. In order to accommodate these students, many universities and colleges are offering 
adult students greater numbers and varieties of online classes (Ausburn, 2004). Online classes are taught mostly on 
the Web while hybrid classes combine the traditional face-to-face class meetings with online course sessions. A 
class is web-enhanced if its traditional classroom course uses the Web to enhance the course’s content (Shelly, 
Cashman, Gunter, & Gunter, 2008). The Sloan Consortium, a nonprofit group devoted to integrating online learning 
into mainstream higher education, reported that 4 million students in the United States took at least one online 
course in 2007 (“Tech Smart,” 2009). Just three years earlier, the reported enrollment for online courses was 2.3 
million. According to Hughes and Hagie (2005), there is no evidence that enrollment rates in online courses will 
slow. 

The emergence of online education, brought about by the advent of the Internet (Robinson & Hullinger, 
2008) and its subsequent growth, has been driven by student demand (Tabatabaei, Schrottner, & Reichgelt, 2006). In 
online classes, students and instructors do not see each other face-to-face, but interact through technology assisted 
components such as e-mail and discussion boards. Course content can also be conveyed through video, 
photographic, and/or audio arrangements (Hughes & Hagie, 2005). Online education is flexible in that it is available 
7 days per week, 24 hours per day in an asynchronous environment (Osman, 2005). To be successful in the cyber 
schoolhouse, students must possess basic technology skills and access to the Internet. 

Need for the Study 

While studies have investigated whether or not online learning is as effective as face-to-face learning in 
achieving learning outcomes (Robinson & Hullinger, 2008), a search of the literature revealed no studies that 
compared the actual time spent on assignments among the three delivery models: online, face-to-face, and hybrid 
learning environments. 

Review of the Literature 

Online learning solves the problem of working professionals, single parents, military personnel and those in 
rural areas who cannot enroll in traditional higher education classes due to their schedules and physical location. 
While traditional face-to-face classes have a set time and place, the online classroom is available 24 hours per day 
(Mentzer, Cryan, & Teclehaimanot, 2007). Many studies have been conducted comparing online and face-to-face 
classrooms (McCray, 2000; Mentzer et al.,2007). Keegan (as cited in Simonson, Smaldino, Albright & Zvacek, 
2000) states that the online learner and the local learner in the face-to-face classroom should have experiences which 
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are of equal value even though these experiences might be quite different. While research supports (Dennis et al., 
2006) that student learning is equal in online and face-to-face classes, it has been noted that students in online 
classrooms are less satisfied with their learning experiences (Mentzer et al., 2007). Miller, Rainer, and Corley (as 
cited in Mentzer, et al., 2007) found that online students reported that the negative aspects of an online class 
included procrastination, poor attendance, and a sense of isolation. 

Other researchers suggest that online students may be of a different “ilk” than students who enroll in a face- 
to-face class (Daughenbaugh, Surrey & Istam as cited in Mentzer et al., 2007). Students who enroll in online classes 
may have other obligations in addition to attending school, a factor that could result in their achieving lower grades. 
A second factor could be that online students who succeed in the cyber classroom view online education as 
advantageous due to its greater flexibility and convenience (Tabatabaei, Schrottner, & Reichgelt, 2006). 

Mentzer, Cryan, and Teclehaimanot (2007) compared learning outcomes and student perceptions of and 
satisfaction with two sections of the same course. Students were randomly assigned to either an online or a 
traditional face-to-face class, taught by the same instructor. While test performance was equivalent for the two 
groups, over-all grades were higher in the traditional face-to-face class due to the many incomplete assignments in 
the online class. In addition, students in the face-to-face class rated the course statistically higher than students in the 
online class. The researchers concluded that the encounter with the instructor resulted in greater motivation for the 
students in the face-to-face class. 

Online learning demands that students make a hefty time commitment as well as take more responsibility 
for their own learning (Simonson, Smaldino, Albright, & Zvacek, 2000). In one study (Bambara, Harbour, Davies, 

& Athey, 2009) students reported that their online classes required more time than traditional face-to-face classes 
because they had to teach themselves rather than relying on the teacher for instruction. Moreover, some reported that 
online classes required more time and effort than they had planned to invest in preparing, organizing, and managing 
their resources. 

Face-to-Face Classrooms 

Traditional education can be defined as education that “takes place at the same time in the same place” 
(Simonson, Smaldino, Albright, & Zvacek, 2000, p. 7). The practice of teaching adults in face-to-face classrooms 
has its roots in antiquity and could be said to have begun with philosophers and religious leaders such as Socrates, 
Plato, and Jesus. Socrates used a dialectical question-and-answer approach that is now called the Socratic method 
(Ozmon & Craver, 1999). Although Socrates left no written record of his methods, Plato, his disciple, illustrated the 
Socratic method through many written dialogues which scholars today refer to as Platonic philosophy. To use the 
Socratic dialectic effectively, students needed training, experience, and maturity (Ozmon & Craver, 1999). Jesus, 
too, was a master of the Socratic question-and-answer technique (Price, 1946). This traditional face-to-face learning 
environment provides a synchronous mode of communication which fosters student engagement and motivation as 
well as immediate feedback from instructors (Dennis et al., 2006). 

Online Classrooms 

Online classes, obviously, are not rooted in antiquity. Rather, the online classroom has evolved from the 
concept of distance education which occurs when “the learner is physically separated from the teacher” (Rumble as 
quoted in Simonson et al., 2000, p. 7). The earliest distance education dates to 1833 when a Swedish newspaper 
offered a correspondence course in composition. This new educational phenomenon, the correspondence course, 
spread first to England and then to Germany before finally arriving in Boston in 1873. When the University of 
Chicago was founded in 1890, a university extension was created with correspondence study as one of its five 
divisions (Simonson et al., 2000). The next innovation in distance learning came in the form of audio recordings for 
the blind. During the 1920s, technology expanded to include radio, and by the 1930s some universities were 
experimenting with television, although it was not until 1 95 1 that the first university courses via television were 
offered for college credit at Western Reserve University (Simonson et al., 2000). As satellite technology became 
more sophisticated in the 1960s and more cost effective in the 1980s, instructional television spread to such remote 
areas as Appalachia and Alaska. The development of fiber-optic communication systems in the late 1980s and the 
early 1990s allowed communication via television to become two-way, with high-quality audio and video systems 
(Simonson et al., 2000). Distance education as we know it today is a result of computer mediated communication 
made possible through the Internet. Today, hundreds of courses are offered totally online, with students never 
meeting the instructor in person. 

Online education requires students to possess basic technology skills as well as access to the Internet. The 
flexibility of online learning allows learners to study at their convenience because the online classroom is available 
24 hours a day, 7 days per week in an asynchronous environment (Osman, 2005). 

Hybrid Classrooms 
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Since learning is a social practice (Knights, 1993), the communal dimension of learning should not be 
undervalued. Knights points out that groups can exert a powerful influence to advance learning. According to 
London and Sessa (2006), a group can be defined as “two or more people who interact with each other to perform a 
task” (p. 304). The outcome of learning in groups can be multiple perspectives and idea-sharing. One disadvantage 
that online learners note (Rovai & Jordan, 2004) is that online learning is often a solitary experience. Many online 
learners miss the personal interaction with peers and instructors (Bambara et al., 2009). Hybrid classes could be the 
solution to the problem. Hybrid classes represent the thoughtful integration of face-to-face and online learning 
delivery models that blend the strengths of each into a “unique learning experience congruent with the context and 
intended educational purpose” (Garrison & Vaughan, 2008, p. 5). Hybrid learning is any “combination of self- 
paced, instructor-led, distance, and classroom delivery with various digital and print form factors” (Bonk & Graham, 
2006, p. 97). It could be that hybrid classrooms may offer today’s student the best of both worlds. Just as hybrid cars 
combine electricity and gasoline to produce a more efficient mode of transportation, hybrid classrooms combine 
traditional face-to-face classroom instruction and technology-assisted instructional techniques with the goal of 
creating a more efficient learning environment that increases student engagement. Students in a hybrid classroom 
have the advantage of personal contact with the instructor as well as the flexibility of online delivery. Garrison and 
Vaughan (2008) posit that education is “best experienced in a community of inquiry” (p. 26). Rovai and Jordan 
(2004) found that students in hybrid courses experienced “stronger feelings of community” (p. 10) than did students 
in either face-to-face or online classes. Hybrid education is the new buzz word in education circles — especially in 
higher education circles. Hybrid courses are changing the way instruction is delivered and the way that students 
learn (Ausburn, 2004; Buzzetto-More & Sweat-Guy, 2006; McCray, 2000; Reynard, 2007). The modern teaching 
and learning environment demands new teaching and learning strategies (Rovai & Jordan, 2004). Rovai and Jordan 
also note that a discussion that is started in a face-to-face class can be extended to an online discussion that permits 
time for reflection. 

Purpose of the Study 

The purpose of this study was to compare the actual amount of time graduate students enrolled in face-to- 
face, online, and hybrid classes spend on assignments in order to determine which delivery model is the most 
efficient. 

Hypotheses 

This study was designed to allow the researcher to test the following hypotheses. 

Research Hypothesis I: Graduate students enrolled in an online course will spend more time in class 
participation than graduate students enrolled in a traditional face-to-face classroom. 

Research Hypothesis II: Graduate students enrolled in an online course will spend more time in class 
participation than graduate students enrolled in a hybrid course. 

Null Hypothesis: There will be no difference in the amount of time spent in class participation of graduate 
students in an online class when compared with graduate students in a traditional face-to-face class. 

Research Question 

What is the difference in the time spent preparing assignments for graduate students enrolled in an online class when 
compared with graduate students enrolled in a traditional face-to-face class or a hybrid class? 

Theoretical Framework of the Study 

The theoretical framework for this study, andragogy, is a learning theory that focuses on process (Knowles, 
Holton, & Swanson, 1998), making it especially pertinent to instruction that integrates technology into the learning 
process. Knowles et al point out four characteristics of adult learners that should inform the design of instruction that 
incorporates technology. First, because adults have a deep need to know why they should learn something before 
they invest their time and energy, instructors should explain the purpose of technology in the instructional design. 
Next, instructors should consider the interests of the learners because adults learn best those things they must know 
in order to perform tasks that are relevant to them. Third, the instructor should find out the background experiences 
of the learners in order to give them choices based on those prior experiences. Finally, since adults are self-directing 
and dislike having decisions imposed on them, instructors should allow adult learners to figure things out for 
themselves. Instructors, in short, should become facilitators rather than directors of learning. 

Methods 

Participants 

Participants in this study consisted of 10 female students enrolled in an online class, 6 female and 4 male 
students enrolled in a face-to-face class, and 21 female and 6 male students enrolled in a hybrid learning class. 
Participants, ranging in age from 23-57, were teachers pursuing a master’s degree in education. The instructor was 
the same for the three classes. The course, a graduate course in the social and philosophical foundations of 
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education, consisted of the same course content for all participants. The difference among the three groups was the 
delivery model. 

Instrumentation 

The survey instrument, the Time Measurement Inventory, was created by the researcher, pilot tested by 18 
students in a graduate research class, and revised based on feedback from the participants in the pilot test. 
Participants gave 100% approval for the revised instrument. The following questions were analyzed by ANOVA to 
determine statistical differences. 

1 . How long did it take you to interview the retired teacher? 

a. 15-30 minutes 

b. 31-45 minutes 

c. 46-60 minutes 

d. 61-75 minutes 

e. More than 75 minutes 

2. How long did it take you to write your biography? 

a. 1 hour or less 

b. 1 hour - 1 14 hours 

c. 1 Vi hours - 2 hours 

d. 2 hours - 2 14 hours 

e. More than 2 14 hours 

3. How long did it take you to read two biographies written by students, write, and post your response to the 
biographies? 

a. 15-30 minutes 

b. 31-45 minutes 

c. 46-60 minutes 

d. 61-75 minutes 

e. More than 75 minutes 

4. To what degree was the value of this assignment to you in understanding the social and philosophical views of 
a previous era? 

a. Very valuable 

b. Valuable 

c. Somewhat valuable 

d. Very little value 

e. No value 

5. How often do you work on your online assignments? 

a. Weekly 

b. Every fourth day 

c. Every other day 

d. Daily 
Data Collection 

Data were collected through a quantitative/qualitative survey administered upon the completion of a 
specific assignment. Of the 13 online students, 10 returned the survey. Of the 10 students in the face-to-face class, 

10 students returned the survey. Of the 27 students in the hybrid class, 27 returned the survey. To protect the 
anonymity of the online students, participants e-mailed their surveys to a coworker of the researcher who removed 
any identification from the e-mails. When all surveys were returned, the coworker placed the surveys in a plain 
envelope before giving them to the researcher. The face-to-face and hybrid surveys were administered by a student 
in the class (after the researcher left the room) who placed the surveys in a plain envelope before giving them to the 
researcher. While the assignment was basically the same for the three groups (interview a retired teacher and 
compose a biography of the teacher according to guidelines provided), the reporting format of the biography was 
tailored to the classroom delivery model. The survey instrument, the Time Measurement Inventory, was designed to 
determine the actual amount of time students spent completing the assignment and to measure the value of the 
assignment to the student. 

Data Analysis 

Quantitative data gathered through surveys were analyzed by ANOVA in order to measure whether or not 
there were significant differences (Creswell, 2008) among the group means. Since there were more than two groups, 
t-tests were performed to determine the specific pairwise differences in the event of a significant ANOVA f-test. 
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Global type 1 error was controlled at the = 0.05 level. Responses to open-ended questions were analyzed 
according to like themes (Miles & Huberman, 1994). 

Results 

The results for questions 1-4 were not statistically significant, although these results are accompanied by 
weak power. It is possible, therefore, that a re-test of questions 1-4 with a larger data set would provide either 
significant results or increased statistical power for null effects. 

There was, however, a significant mean difference ( = 0.05) for question 5. Specifically, the online 

students reported that significantly more time was required to complete their work (mean = 4.0) than students in the 
blended group (mean = 2.2). There were no data available for the face-to-face students for question 5. A Levene’s 
Test for Equality of Variance failed to reject the assumption of equal variance at a level of = 0.05, which supports 
the assumption of equality of variance for ANOVA. Table 1 reports the results of the ANOVA test for question 5. 





N 


stdev 


mean 


Flybrid instruction 


27 


1.43 


2.2 


Online instruction 


10 


1.05 


4.0 


Face-to-face instruction 






n/a 



Table 1: ANOVA for Q5 

Discussion 

Results of data analysis support the benefits of hybrid classes as far as the time spent on class preparation. 
Of the 27 students in the hybrid class, 20 expressed that they preferred the hybrid classroom, 4 indicated that they 
preferred the face-to-face classroom, while the remaining 3 expressed a preference for online learning. While the 
reasons varied somewhat, students preferring face-to-face classes all reported that they preferred face-to-face classes 
because of the personal interaction with teachers and other students. Of those in the hybrid classes who preferred the 
online delivery model, all three students noted that in online classes they could work at their own pace. Two 
mentioned the savings in gas, and one noted that online classes left more time for family. One student who 
expressed a preference for online classes did note that online classes took more time. The majority of the students 
(20) in the hybrid class preferred the hybrid mode of delivery. Most noted that they preferred the hybrid classroom 
because they could meet with the instructor and their fellow students (more personal) yet have time to work 
independently as well, giving them more time to spend with family. Several also commented that the hybrid class 
offered more flexibility in scheduling and more freedom to work “on my own.” 

Limitations 

Limitations of this study include a small sample size (47 total participants) and a limited population 
(graduate students). Replication of this study with a larger sample is recommended as a larger sample size could 
offer increased statistical power. Future studies could include undergraduate classes as well as graduate classes. This 
research could be further extended by including a wider range of online instructional strategies that would facilitate 
diverse learning needs and learning preferences. 

Conclusions 

The distinguishing feature of this study is its focus on the actual time students in face-to-face, online and 
hybrid classes spend in class preparation. Findings from this study will provide students with a guide to help them 
choose the most efficient delivery model for their particular academic needs. Furthermore, this study contributes to 
the emerging knowledge base regarding the effectiveness of the hybrid classroom. This study can facilitate faculty in 
designing courses that meet the diverse needs of adult learners who are nontraditional in that they often work full- 
time and have family obligations in addition to pursuing higher education (Ausburn, 2004). 

As with any new paradigm that promises to be the wave of the future, it is important to determine whether 
or not the marriage of these two teaching models (traditional face-to-face classrooms and online classrooms) 
maximizes student engagement and fosters student learning in such a manner that the newly created third model is, 
indeed, a new learning model and not simply “more of the same” in a new package. Ausburn (2004) points out that 
hybrid learning is an effective alternative approach because it can combine the best features of each model — the 
flexibility of the 24/7 online classroom, so valuable to many working adults, with the rapport, interaction, and 
support among participants in the face-to-face classroom. Garrison and Vaughan (2008) note that hybrid learning is 
actually a “fundamental redesign and the consideration of new approaches to learning” (p. 26). They further stress 
the importance of integrating the real and virtual communities, pointing out that “face-to-face verbal and online text 
communication are distinct and have the enormous potential to complement each other” (p. 27). Thus, the nature of 
the educational experience can be transformed through the use of direct and mediated communication which can 
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come as a result of the rethinking of the educational approach (Garrison & Vaughan, 2008). The face-to-face 
classroom is collaborative. Discussions that begin in the classroom can be extended in the online classroom because 
the online classroom allows time for reflection and consideration of that discussion, thus giving each participant the 
opportunity to provide a teaching presence (Garrison & Vaughan, 2008). In the words of one student who preferred 
the hybrid classroom, “It is the best of both worlds.” 
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Abstract: Asynchronous online discussions are common in online and blended courses. 
This study examined the impact of online discussions in blended undergraduate courses 
in three disciplines: educational technology, engineering, and English education. Results 
showed that students from all courses were comfortable using online discussions and saw 
them as a way to express opinions and learn course content. However, differences in the 
outcomes for three groups suggest that certain online discussion strategies are more 
useful than others. 



Introduction 

Online and blended forms of learning are expanding rapidly in U. S. higher education. The sixth annual 
Sloan Consortium national survey of online learning in the U. S. (Allen & Seaman, 2008) found that over 3.9 
million college and university students, about 20% of all higher education students, took at least one online course in 
the fall of 2007. Online learning enrollments in 2007 grew 12.9% compared to the previous year, a rate far in excess 
of the growth of the overall higher education student population. Blended or hybrid courses, those that combine 
elements of traditional face-to-face learning with elements of online learning, are also growing in popularity and are 
offered in proportions similar to fully online courses (Allen, Seaman, & Garrett, 2007). 

Although many instructional strategies can be employed to foster student learning online (cf. Bonk & 
Zhang, 2008), one of the most widely used instructional approaches is the asynchronous online discussion. 
Asynchronous online discussions have been an integral part of many computer-mediated courses since the inception 
of this form of teaching and learning (Harasim, 1990; Hiltz & Turoff, 1993). In online courses, asynchronous 
discussions replace in-class discussions, while in blended or hybrid courses they can extend face-to-face discussions 
and provide another way for students to interact with each other and with the content. Today, numerous tools for 
conducting online discussions are available including those built into commonly used course management systems 
such as Blackboard, Angel, and Moodle. Although asynchronous online discussions are commonplace, relatively 
little has been written about their use in different disciplinary contexts. This study examined the use and perceived 
impact of online discussions in blended undergraduate courses in three different disciplines: educational technology, 
engineering, and English education. 

Background 

Developing group interaction and problem-solving skills is a goal in both education and the corporate 
world (Dundis & Benson, 2003). In corporations, employees must be able to communicate and solve problems 
within a team context. In education, standards (cf. National Science Education Standards, National Research 
Council, 1996) call for teachers to develop communities of learners, nurture collaboration among students, and 
structure and facilitate formal and informal discussions to promote student learning. Rising Above the Gathering 
Storm , the oft-cited report from the National Academies (2007), suggests that in order to create an environment and 
culture that support innovation in the U.S., our organizations must value social factors including “collaboration, 
communication, the treatment of multiple viewpoints” and utilize technological factors such as “access to high- 
speed computing and communications” (p. 417). 

Asynchronous online discussions are one way to utilize computer-mediated communication to promote 
student collaboration and learning in the educational process. In online courses, asynchronous discussions serve as a 
stand-in for the dialogue and interchange typical of most face-to-face courses. In blended or hybrid courses, online 
discussions can extend face-to-face discussions beyond the confines of the classroom to increase students’ 
engagement with the content and with one another. Students tend to respond positively to the asynchronous 
discussion format because it allows them to participate at their convenience, gives them time to think about and 
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consider points made by peers before responding, and keeps a written record of all contributions for review and 
reflection (Tiene, 2000). Students perceive online discussion to be more egalitarian than traditional classroom 
discussions (Harasim, 1990), and online discussions create a sense of social presence that helps to create community 
online (Gunawardena & Zittle, 1997; Rourke, Anderson, Garrison & Archer, 2001). According to Palloff and Pratt 
(1999), "The learning community is the vehicle through which learning occurs online. It is the relationships and 
interactions among people through which knowledge is generated" (p. 15). 

This emphasis on a community of learners in the educational process mirrors the workplace where teaming, 
collaborative problem solving, and group inquiry, often conducted virtually, are becoming the norm. As technical 
workplaces have become increasingly computer-centered, virtual collaboration through computer networking has 
become an essential skill for success in the 21 st century (Bourne, Harris, & Mayadas, 2005; Johnson, Suriya, Yoon, 
Berrett, & LaFleur, 2002). Online discussions have the potential to assist students in the construction of knowledge 
and serve as a scaffold that allows for multiple perspectives, negotiation of meaning, and an understanding of 
knowledge gaps a learner may possess (Haavind, 2006). A meta-analysis of the effects of distance education 
compared to classroom instruction found that students using media that supported asynchronous discussion in 
distance education significantly outperformed students in the traditional classroom (Lou, Bernard, & Abrami, 2006). 

However, there are still many questions about the use of online discussions. For one thing, the use and 
efficacy of online discussions may differ across disciplinary contexts. This study, part of a larger project on the use 
of peer feedback in online discussions, investigated students’ perceptions of the use of online discussions as part of 
hybrid/blended courses and their effect on students’ motivational orientations and use of learning strategies in three 
different disciplinary contexts: educational technology, engineering, and English education. 

Methods 

This study was conducted in the fall of 2008 at a large Midwestern university. Participants were students 
enrolled in three undergraduate courses: an introductory educational technology course, an engineering digital 
systems design course, and an English education methods course. The courses were required for students in their 
respective disciplines. Students in the educational technology course were mostly freshmen and sophomores, and, 
on average, their prior experience with online discussions was limited (40% had no prior experience and another 
29% had only one prior experience in a course with online discussions). Similarly, the engineering students were 
mostly sophomores with limited prior experience (36% had no prior experience and another 31% had only one prior 
experience). In contrast, the English education students were seniors nearing the end of a teacher preparation 
program with more experience using online discussions (78% had participated in three or more courses that had used 
online discussions). 

The students in each course engaged in three online discussions related to course content as part of a 
blended approach that supplemented regular course activities during a 16-week semester. The educational 
technology and engineering courses each had face-to-face lecture and laboratory components; students’ laboratory 
sections served as the grouping for the online discussions. In the English education course, the entire class 
participated in the online discussions as a supplement to regular face-to-face class meetings and discussions. 
Students in the educational technology course participated in discussions on learning theories, millennial students, 
and plagiarism. Students in the engineering course participated in discussions focused on homework problems on 
course concepts and exam preparation strategies. Students in the English education course participated in 
discussions on teacher identity, teaching literature, and action research. All online discussions were hosted in the 
discussion forum of Blackboard Vista. 

To assess students’ perceptions of the online discussions, at the end of the semester, students in all three 
courses were asked to complete an online survey that included questions about perceptions of the online discussions 
and their perceived impact. Completed surveys were received from students in the educational technology course 
(n=219), the electrical engineering course (n=103), and the English education course (n=18). Closed and open- 
ended items assessed students’ comfort and confidence using the online discussions, advantages and limitations, and 
effects of the online discussions. Results of closed-ended items were tabulated, and outcomes for the three courses 
were compared to identify any differences in the responses of the three groups of students. Open-ended survey 
responses were analyzed using a simple pattern-seeking method to gather qualitative responses that were used for 
triangulation of the quantitative results. To assess participating students’ motivational orientations and use of 
learning strategies, the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & 
McKeachie, 1991) was administered on a pretest-posttest basis at the beginning and end of each course. Completed 
questionnaires were received from students in the educational technology course (n=172), the electrical engineering 
course (n=93), and the English education course (n=20). The MSLQ subscale scores were tabulated and pretest- 
posttest scores were compared within groups to look for evidence of changes. 
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Results and Discussion 
Students’ Perceptions of Online Discussions 

End-of-semester survey items were used to examine students’ perceptions of the online discussions in each 
of the three classes. Table 1 presents the means and standards deviations for end-of-semester survey items dealing 
with comfort and confidence related to participating in the online discussions, collaboration/teamwork, and feedback 
from peers and instructors. Items were assessed on a 5-point Likert-like scale, from 1-low to 5-high, except for the 
collaboration/teamwork items, which were based on a 4-point scale. Means were calculated for each item. 

Table 1 



Comparison of Online Discussion Survey Means from the Three Courses 



Survey Item 


Ed Tech 
Mean (SD) 

(n=219) 


Engineering 
Mean (SD) 

(n=103) 


English Ed 
Mean (SD) 

(n=18) 


Comfort/Confidence 
Comfort using online discussion tool 


3.80 (1.06) 


3.32 (1.21) 


4.39 (0.85) 


Comfort contributing to online discussions 


3.72 (1.07) 


3.12 (1.11) 


4.28 (0.75) 


Comfort commenting on others’ contributions 


3.59(1.11) 


2.96 (1.06) 


3.94 (1.00) 


Confidence in ability to contribute relevant ideas 


3.84 (0.97) 


3.16 (1.14) 


4.28 (0.83) 


Confidence in ability to benefit from discussions 


3.41 (1.05) 


3.15 (1.14) 


3.00 (0.91) 


Collaboration/T earn work " 

Level of collaboration with peers as a result of 
online discussions 
Feeling of teamwork among peers 


3.11 (0.72) 
2.86 (0.83) 


3.35 (0.79) 
2.88 (0.86) 


2.94 (1.06) 
2.72 (0.96) 


Feedback 

Usefulness of feedback received from peers 


3.18 (0.87) 


3.21 (0.75) 


2.78 (1.00) 


Helpfulness of TAs’ participation in online 
discussions 


3.36 (1.02) 


3.06 (1.36) 


2.11 (1.32) 



* Collaboration/Teamwork items used a 4-point scale; all other categories used a 5-point scale 



Results related to comfort with and confidence in using the online discussions suggest that students in all 
three classes were relatively comfortable participating in the online discussions. Means for “Comfort using online 
discussion tool” were above 3 (neutral) for all three classes. The English education students had the highest comfort 
mean, which is not a surprise given that these students were mainly seniors who were more experienced with online 
discussions and were more familiar with their classmates than students in the other two classes. Means for “Comfort 
contributing to online discussions” and “Comfort commenting on others’ contributions” were also above 3 for the 
educational technology and English education students; however, means for the engineering students were close to 3 
(neutral), suggesting that these students were somewhat less comfortable contributing to the discussions and 
commenting on others’ discussion posting. This may be related to the nature of the engineering class, which was not 
particularly discussion-oriented, and the online discussions in that class, which focused mainly on solving problems 
related to digital circuit design. The technical nature of these online discussions may have led students to be less 
comfortable making comments and less confident in their own contributions. Indeed, a similar response pattern is 
seen for “Confidence in ability to contribute relevant ideas” where the engineering students rated their confidence 
lower than students from the other two classes. On the final comfort/confidence item, “Confidence in ability to 
benefit from discussions,” the educational technology students demonstrated the highest level of confidence, while 
students in the other two classes gave more neutral responses, on average. The relatively lower means on this item, 
particularly for the engineering and English education students, suggest that students were uncertain about the value 
of the online discussions. 

Students in all three classes tended to give positive ratings to the effects of the online discussion on their 
“Level of collaboration with peers as a result of online discussions” and “Feeling of teamwork among peers.” Mean 



137 





scores clustered around 3 on a 4-point scale (Table 1). This suggests that the students in all three classes perceived 
the online discussions as promoting collaboration with peers. The engineering students gave the most positive 
responses to these items. This is consistent with findings in a previous project semester (Lehman, Richardson, 
Ertmer, Newby, & Campbell, 2009) and may reflect the collaborative problem-solving culture in engineering. In the 
engineering course, students were used to working together to solve homework problems, and the online discussions 
helped to facilitate this process which may have contributed to a greater sense of collaboration and teamwork 
resulting from the online discussions among the engineering students. 

For the “Usefulness of feedback received from peers” and “Helpfulness of TAs’ participation in online 
discussions” items, mean scores for both the educational technology students and engineering students were above 3 
on a 5-point scale, indicating somewhat positive responses. However, the means for the English education students 
were less than 3 indicating somewhat negative responses. These differences in response patterns likely reflect how 
the discussions were used in each course. In the educational technology course, students participated in well- 
structured discussions in which peers made contributions and teaching assistants (TAs) were active discussion 
facilitators. Not surprisingly, students rated instructor participation the highest in this course. In the engineering 
course, discussions were mostly peer-driven with occasional TA involvement to clarify a problem solution, for 
example. In the English education class, only the course instructor was involved; there were no TAs. As a 
consequence, students in the English education class gave the lowest rating to helpfulness of TA participation. 

Student Perceptions of Outcomes 

Students responded to survey items about the outcomes of participation in the online discussions including 
the perceived effect on their learning, attitudes toward peer learning, whether they had become better acquainted with 
classmates, and met with classmates outside of class. Results are shown in Table 2. 

Table 2 



Frequencies of Responses Related to Learning Outcomes by Course 



Learning Outcome 


Ed Tech Students 


Engineering Students 


English Ed Students 






(n=219) 






(n=103) 






(n=18) 






Yes 


No 


Unsur 


Yes 


No 


Unsur 


Yes 


No 


Unsur 


Perceived differences in 






e 






e 






e 


learning 


34.7% 


36.5% 


28.8% 


32.0% 


46.6% 


21.4% 


11.1% 


72.2% 


16.7% 




Pos 


Neg 


Neut 


Pos 


Neg 


Neut 


Pos 


Neg 


Neut 


Attitudes toward peer 
learning 


45.2% 


20.6% 


34.3% 


37.9% 


11.6% 


50.5% 


33.3% 


11.1% 


55.6% 




Yes 


No 


Unsur 


Yes 


No 


Unsur 


Yes 


No 


Unsur 


Better acquainted with 






e 






e 






e 


classmates 


18.7% 


68.0% 


13.2% 


14.6% 


69.9% 


15.5% 


33.3% 


61.1% 


5.6% 




Yes 


No 




Yes 


No 




Yes 


No 




Met with classmates outside 
class 


18.9% 


81.1% 




40.8% 


59.2% 




66.7% 


33.3% 
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A minority of students from all three courses (34.7% of educational technology students, 32.0% of 
engineering students, and 11.1% of English education students) felt that the online discussions made a difference in 
their learning. This is consistent with the responses to the “Confidence in ability to benefit from discussions” item 
reported above (see Table 1). Although some students perceived a learning value in the online discussions, more 
students did not or were neutral. However, a greater proportion of students reported a positive attitude toward peer 
learning (45.2% of the educational technology students, 37.9% of the engineering students, and 33.3% of the English 
education students). This is consistent with the “Level of collaboration with peers as a result of online discussions” 
item responses reported above (see Table 1). Relatively few of the students in any of the classes felt that they became 
better acquainted with their classmates through the online discussions. However, two-thirds of the English education 
students and two-fifths of the engineering students met classmates outside of class compared with less than one-fifth 
of the educational technology students. This is likely not so much the influence of the online discussions as it is a 
reflection of the cohesiveness of the students, in the case of the small English education class, or an outgrowth of the 
collaborative problem-solving culture in engineering, in the case of the engineering class where students often 
worked together on homework and to prepare for examinations. 

Motivation and Learning Strategy Outcomes 

To assess whether the online discussions might influence students’ motivational orientations and use of 
learning strategies, we employed a pre- and post-test administration of the Motivated Strategies for Learning 
Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1991). Results are shown in Table 3. For most of 
the subscales of the MSLQ, students’ motivational orientations and use of learning strategies tended to show little 
change or declines from beginning to end of semester. 

Students in the English education class showed the least changes from pre- to post-administration of the 
MSLQ. These students increased significantly on only one subscale, self-efficacy. Students in the engineering 
course showed declines on a number of the MSLQ subscales. The educational technology students showed a mixture 
of gains and declines on various subscales. Notable gains were observed in self-efficacy, intrinsic motivation, and 
peer learning. The MSLQ changes observed probably do reflect characteristics of the individual courses. For 
example, the gain in self-efficacy in the English education course makes sense when one considers that this was a 
methods course that was preparing students for their student teaching experience. The educational technology course 
had a popular group project that may have contributed to students’ gains in self-efficacy, intrinsic motivation, and 
peer learning. The declines on several scales in the engineering course may be a function of the fact that this was a 
difficult beginning-level digital electronics course. However, it is doubtful whether any of these shifts can be 
attributed to the influence of the online discussions, which were only a small part of each of these three courses. In 
addition, the MSLQ post-administration occurred at the end of the semester, quite a while after completion of the 
online discussions. Thus, it seems likely that these results are confounded and so not particularly meaningful in 
assessing the impact of the online discussions. 
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Table 3 

MSLQ Pre- and Post-Test Results by Course 





MSLQ Scale 


n 


Pretest Mean 
(SD) 


Posttest Mean 
(SD) 


Paired Samples 
t-value 




intrinsic motivation 


172 


4.69 (0.91) 


4.91 (0.89) 


3.12 ** 




extrinsic motivation 


172 


5.59(0.91) 


5.44 (1.06) 


-1.72 




task value 


172 


5.13 (0.97) 


5.15 (1.01) 


0.16 




control beliefs 


172 


5.21 (0.95) 


5.34 (0.93) 


1.66 


E 


self-efficacy 


172 


5.24 (1.02) 


5.84 (0.91) 


7 Q9 *** 


d 


test anxiety 


172 


4.12 (1.19) 


3.85 (1.36) 


-2.93 ** 




rehearsal 


172 


4.40 (0.99) 


4.42 (1.30) 


0.17 


T 


elaboration 


172 


4.72 (0.93) 


4.54 (1.12) 


-1.89 


e 


organization 


172 


4.24 (1.08) 


4.19 (1.38) 


-0.46 


c 

v 


critical thinking 


172 


4.02 (1.00) 


4.22 (1.23) 


2.14 * 


h 


self-regulation 


172 


4.24 (0.80) 


4.32 (0.90) 


1.21 




time study 


172 


5.11 (0.87) 


4.55 (0.88) 


-7 78 *** 




effort regulation 


172 


5.01 (1.10) 


4.78 (0.96) 


-3.02 ** 




peer learning 


172 


3.01 (1.10) 


3.72 (1.52) 


6.24 *** 




help seeking 


172 


4.14(1.10) 


4.18(1.09) 


0.41 




intrinsic motivation 


93 


5.30 (1.02) 


4.94(1.16) 


-3.39 ** 




extrinsic motivation 


93 


5.72 (0.94) 


5.31 (1.14) 


_4 12 *** 


E 


task value 


93 


5.73 (1.01) 


5.25 (1.25) 


_4 12 *** 


n 


control beliefs 


93 


5.75 (0.88) 


5.34 (1.09) 


-3.60 ** 


g 


self-efficacy 


93 


5.64 (0.90) 


5.05 (1.09) 


-4 78 *** 


i 


test anxiety 


93 


3.97 (1.34) 


4.08 (1.27) 


0.98 


n 


rehearsal 


93 


4.27 (1.30) 


4.28(1.31) 


0.08 


e 


elaboration 


93 


4.99 (1.06) 


4.81 (1.02) 


- 1.68 


e 


organization 


93 


4.46 (1.36) 


4.35 (1.33) 


-0.73 


r 


critical thinking 


93 


4.54 (1.38) 


4.46(1.22) 


-0.56 


i 


self-regulation 


93 


4.47 (0.88) 


4.43 (0.84) 


-0.50 


n 


time study 


93 


5.01 (0.84) 


4.51 (1.06) 


-4 78 *** 


g 


effort regulation 


93 


5.23 (0.98) 


4.68 (1.04) 


- 5.45 *** 




peer learning 


93 


4.12(1.34) 


3.92(1.41) 


-1.44 




help seeking 


93 


4.21 (1.32) 


3.91 (1.34) 


-2.28 




intrinsic motivation 


20 


5.35 (0.87) 


5.68 (0.80) 


1.64 




extrinsic motivation 


20 


4.79 (0.96) 


4.96 (1.02) 


0.92 




task value 


20 


6.29 (0.63) 


6.08 ( 1 . 00 ) 


-1.07 


E 


control beliefs 


20 


5.28 (0.86) 


5.61 (0.94) 


1.31 


n 


self-efficacy 


20 


6.22 (0.67) 


6.57 (0.52) 


2.36 * 


g 


test anxiety 


20 


3.38 (1.50) 


3.10(1.41) 


-1.18 


1 


rehearsal 


20 


3.93 (1.40) 


3.58 (1.68) 


-1.27 


i 


elaboration 


20 


5.07 (1.02) 


4.82(1.19) 


-0.85 


s 


organization 


20 


3.63 (1.27) 


3.43 (1.33) 


- 0.68 


h 


critical thinking 


20 


4.56(0.91) 


4.55 (1.06) 


-0.06 


T7 


self-regulation 


20 


4.41 (0.67) 


4.34 (0.97) 


-0.34 


h 

A 


time study 


20 


5.48 (0.84) 


5.43 (1.14) 


-0.37 


a 


effort regulation 


20 


5.71 (0.97) 


5.74 (0.91) 


0.16 




peer learning 


20 


3.37(1.19) 


3.43 (1.87) 


0.19 




help seeking 


20 


4.66 (0.84) 


4.66(1.12) 


0.00 



* p < .05, ** p < .01, *** p < .001 
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Advantages and Limitations 

Students also responded to survey items that addressed the perceived advantages and limitations with 
respect to the online discussions. These are summarized in Table 4, which shows the percentages of students who 
identified specific advantages and limitations. Students could select more than one response, so percentages total 
to more than 100%. 

The most commonly cited advantage for both the educational technology and English education students 
was that the online discussions “Made it easier to express opinions and to participate in class discussions.” More 
than 60% of the educational technology students and more than 70% of the English education students identified 
this as an advantage. This is not a surprise given that discussion is a common instructional method in education, 
and online discussions allow all students in the class to readily participate. For the engineering students, the most 
commonly cited advantage was that the online discussions “Flelped me understand the content better.” More than 
half of the engineering students, as well as about half of the educational technology students, identified this as an 
advantage. The engineering course used the online discussions as a way for students to collaboratively work on 
homework problems and exam preparation strategies, and so it is not unexpected that the students would perceive 
advantages in learning the content. In the case of the educational technology course, the online discussions 
focused on several important topics in the course and so many of these students also perceived that the discussions 
were beneficial for learning course content. More than a third of both the educational technology and engineering 
students also cited “Motivated me to study the course materials or other related topics/content” as an advantage, 
whereas half of the English education students cited, “Flelped me get better acquainted with my classmates.” 

The limitation most commonly cited by all three groups of students was that “It was hard to remember to 
do it.” Nearly half of educational technology and engineering students and almost three-quarters of the English 
education students cited this as a limitation. Many of the students in these classes had relatively little prior 
experience with online discussions and were unaccustomed to this type of class participation, so it was easy for 
them to forget about this outside-of-class-time commitment. In hybrid or blended courses such as these, online 
discussions are an “extra” rather than an essential means of communication, so students may not perceive them as 
really important. Students also felt that “It took too much time;” this limitation was cited by about a quarter of the 
students in each of the classes. Both the educational technology and engineering students had some uncertainty 
about what to post and who was right/correct. These students, who were focused on content learning, seemed to 
be more concerned about being correct and knowing what was correct, something that was not always clear in the 
give and take of the online discussions. For the English education students, and to some extent the educational 
technology students, another limitation was that “It was hard deciding what score to give my peers;” this response 
was a result of the use of a peer feedback tool in the online discussions. Students were unfamiliar with the process 
of giving feedback to their peers, and so they were uncertain what they should do. 

Students elaborated on the advantages and limitations of the online discussion in their responses to open- 
ended questions on the survey. Getting to learn from and share ideas with peers was a common theme among 
students who perceived the discussions as advantageous. As one educational technology student stated, “We don't 
get a chance to really discuss issues in class so this was a way for me to see what my classmates really thought.” 
Another educational technology student noted, “It helped me also understand what my other classmates were thinking 
and that made me see where I was in the class and could compare my thoughts with their thoughts.” An engineering 
student made a similar comment, saying, “I can see that there could be an advantage in being able to discuss a certain 
topic with classmates you would otherwise not speak to in lecture.” For the educational technology and engineering 
courses, where large lectures made in-class participation difficult, the online discussions offered a convenient 
alternative way to share ideas with classmates. This was less of an advantage for the English education course, which 
was small and composed of a group of students that already knew one another fairly well. Flowever, even one of the 
English education students commented that the online discussions, “Gave me an extra chance to get out my thoughts 
and ideas, or to elaborate on things we discussed in class.” 

Flowever, a number of the students felt that the online discussions added little to their learning. One 
educational technology student commented, “I don't really think I have noticed any difference in my learning. I have 
always been a hands-on learner, and I actually prefer a face-to-face discussion over an online one. We spend too 
much time in front of a computer as it is.” Another educational technology student said, “These discussions were 
interesting and something new, but I don't think they really changed anything about the way 1 learn.” An engineering 
student noted that many students participated just because they were required to, saying, “I really do not think that 
online discussions are really that helpful as far as learning goes because a lot of times people just get on, which is 
usually kind of a pain, and write just to write rather than something meaningful.” An English education student 
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commented, “I found that the online post was a restatement of what I learned in class and not a continuation or 
expansion of my ideas.” So, for many of these students, the online discussions did not add enough value to what their 
classes already offered. 



Table 4 



Percentages of Students from Each Course Citing Online Discussion Advantages and Limitations 



Advantages and Limitations 


Ed Tech 


Engineering 


English Ed 




Students 


Students 


Students 




(n=219) 


(n=103) 


(n=18) 


Advantages 








Helped me understand the content better 


50.2% 


52.4% 


11.1% 


Motivated me to study the course materials or 


44.3% 


34.0% 


11.1% 


other related topics/content 








Motivated me to spend time studying course 


32.0% 


26.2% 


5.5% 


materials consistently throughout the course 








(rather than cramming for the exam) 








Made it easier to express opinions and to 


61.2% 


41.7% 


72.2% 


participate in class discussions 








Helped me get better acquainted with my 


18.7% 


14.6% 


50.0% 


classmates 








Other 


7.7% 


13.6% 


16.7% 


Limitations 








It took too much time 


25.1% 


22.3% 


27.8% 


It was hard to remember to do it 


47.5% 


48.5% 


72.2% 


It was hard to ask questions or get help 


16.4% 


18.4% 


5.5% 


I was unsure about how to post 


7.3% 


7.8% 


0.0% 


I was unsure about what to post 


26.9% 


48.5% 


11.1% 


I didn’t know how to respond to others’ postings 


28.3% 


14.6% 


27.8% 


I didn’t know who was right/correct 


25.6% 


40.8% 


5.5% 


It was hard deciding what score to give my peers 


28.8% 


9.7% 


44.4% 


Other 


10.5% 


12.6% 


11.1% 



As a final assessment of what students thought of the online discussions, a survey item asked if they were 
the instructor of the course would they continue the use of the online discussions as is, continue use but with changes, 
or discontinue use? Results from this item are shown in Table 5. Only a minority of students, less than one-fourth of 
the English education students and less than one-fifth of the educational technology and engineering students, would 
not continue using online discussions. Thus, a clear majority of the students would favor continuing the use of the 
online discussions either as used in the course or with changes (e.g., increasing or decreasing the number of 
discussions). This suggests the students did see value in this instructional approach. 
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Table 5 



Student responses from the three courses concerning continuation of online discussions in course 





Ed Tech students 
(n=219) 


Engineering students 
(n=103) 


English Ed students 
(n=18) 


Continue as is 


41.1% 


27.2% 


47.1% 


Continue with changes 


42.0% 


54.3% 


29.4% 


Do not continue 


16.9% 


18.5% 


23.5% 



Implications and Conclusions 

Online and blended forms of learning are becoming increasingly important in higher education, and, as a 
result, there is increasing interest in the use of asynchronous online discussions. This study examined the use of 
online discussions and students’ perception of their impact in three undergraduate blended courses in different 
disciplines. Results suggest that there is potential value in online discussions, but there are differences in the utility 
of online discussions across different content areas and challenges in implementing them effectively in 
undergraduate blended course environments. 

The findings of this study showed that even students who are relatively inexperienced with online 
discussions can, over the course of a single semester, become relatively comfortable with this approach and 
confident in their ability to participate in online discussions as part of blended courses. However, the actual learning 
outcomes from participation in online discussions in blended courses and the value that students place on them are 
less obvious. Many students did seem to appreciate the value of online discussions for providing an avenue for 
expressing opinions and learning from peers. This may be of particular value for larger courses, such as two of the 
three in this study, where large lecture sessions limit the ability of students to participate in class. Many students 
also perceived value of the online discussions for helping them to learn course content. This was particularly true in 
the engineering and educational technology courses, which had a greater focus on content learning. However, many 
of the students did not see these benefits. 

While previous research has suggested that students are satisfied with asynchronous online discussions and 
benefit from them (Johnson, 2006), only a minority of students in this study perceived a direct effect on their 
learning. Whereas the use of asynchronous discussions has been found to lead to performance benefits relative to 
traditional classrooms for distance education contexts (Lou, et al., 2006), it may be more challenging to use 
asynchronous online discussions effectively in blended learning courses for undergraduates where online 
discussions are not essential for student-to-student interaction. 

The challenge for instructors of blended courses who wish to use online discussions is to find ways to 
maximize the perceived relevance and/or value of the discussions. According to Xie, Debacker, and Ferguson 
(2006), when students perceive online discussions as relevant, interesting, and enjoyable their value increases. In 
this study, the engineering students put the highest value of the online discussions on content learning. This result 
certainly reflects the way that the discussions were used in the engineering course (e.g., to help students with content 
problems and for exam preparation) as well as the culture in engineering which puts a premium on problem-solving 
and getting the right answer. On the other hand, the English education and educational technology students in this 
study rated the ability to express opinions and participate most highly, and this is consistent with the nature of these 
disciplines. Achieving concordance between purposes of the online discussion and the goals of the course obviously 
is important. So, instructors of blended courses should seek to use online discussions in ways that fit the discipline 
and content. 

Other differences among the courses in this study also suggest important considerations for integrating 
online discussions. For example, in the educational technology course TAs were active discussion facilitators, and 
students in that course rated this aspect more highly than students in the other courses. According to Talient- 
Runnels, et al. (2006), instructor participation and scaffolding is important for effective learning from online 
discussions. So, efforts to use online discussions in blended courses should give consideration to how the 
discussions are facilitated for best effect. 

For all of the courses, the biggest drawback cited by students was that they had difficulty remembering to 
participate in the online discussions. This is a particularly revealing finding that highlights an important difference 
between fully online and blended courses. In fully online courses, online discussions are typically the central 
vehicle for student-to-student and student-to-instmctor communication. However, in blended courses, online 
discussions are an added form of communication that supplements or complements face-to-face interactions. For 
many of the students in this study, online discussions were perceived as an “extra” rather than as something integral 
to the course and their learning. The challenge for instructors is to develop ways of utilizing online discussions in 
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blended courses that take advantage of the unique features of the approach so that students will not perceive it as just 

one more thing to do. With effective design and implementation, asynchronous online discussions may be an 

effective tool for promoting student learning and collaboration in blended course environments. 
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Abstract 

This study examines the possibilities of an integrated instructional model in a collegiate course using 
the Creative Problem Solving(CPS) with an online support system. An integrated instructional model in a 
collegiate course was developed to enhance the creativity of college students. The principles and guidelines of 
the model were analyzed by the fonnative research methodology. General design principles were identified by 
reviewing the literature, and the theoretical components for the model were extracted. Finally, specific 
guidelines from those design principles and the theoretical components were developed. For this study, the e- 
Leaming contents to explain the major characteristics and thinking tools of the Creative Problem Solving and an 
online support system which can guide the CPS process were developed. The online support system includes 
such convergent and divergent thinking tools as brainstorming, attribute list, hits, PMI, and evaluation matrix. 
Those guidelines were implemented into a college-level course with 33 students in ‘A’ university. The number 
of the final interviewees was 10. The participants' responses to the course satisfaction survey and interviews 
were analyzed to investigate the strengths, weaknesses, and improvements of the model. The strengths include 
‘providing the blended learning environments’, ‘implementing a team project’, and ‘utilizing the CPS support 
system’. The weaknesses and improvements include ‘making participants select appropriate thinking tools for 
the CPS’, ‘explaining the CPS process in details’, ‘proposing a flexible and appropriate schedule for problem 
solving’, and ‘evaluating the final product from the perspective of creativity. Further studies are suggested to 
develop an optimal instructional model using the repetitive formative research methodology and to examine the 
effects of the instructional model by conducting an empirical study are suggested. 

Introduction 

Courses to enhance creativity have recently appeared as relatively new subjects in the college 
environments. Most studies of creativity up to now have focused on the young students ranging from 
kindergarten to junior high school. Flowever, college students, who would be facing the complex problem- 
solving situations in their careers, need to have better creative skills. Regarding the creativity, the Creative 
Problem Solving(CPS) is one of the most frequently referred conceptual models. CPS is understood as a 
structured methodology for enhancing the creative thinking of the individuals and teams. The CPS model 
emphasizes a balance between the divergent and the convergent thinking in every step of the problem solving 
process, and could be deliberately applied to solve the open-ended problems(Puccio, Murdock & Mance , 2005). 

Among the various CPS models that have emerged through several decades of studies, this study was 
based on the CPS version proposed by Treffinger and his colleagues(2000). Their model consists of eight stages: 
'Appraising tasks', 'Designing process', 'Constructing opportunities', 'Exploring data', 'Framing problems', 
'Generating ideas', 'Developing solutions', and 'Building acceptance (Treffinger, Isaksen, & Dorval, 2000). 

The purpose of this study is to develop an integrated instructional model in a college course using the 
Creative Problem Solving with an online support system. The formative research methodology is used to 
improve the principles and guidelines of the model. A model was developed by the following three steps: 
finding out general principles from reviewing the relevant literature; extracting the theoretical components for 
the model by categorizing general principles; and finally, developing a set of specific guidelines for each 
principle from the viewpoint of these theoretical components. 

Theoretical Background 

Creative Problem Solving 

Creative Problem Solving is a model to solve problems in the problematic circumstances by the 
repetitive use of divergent and convergent thinking(Lee & Lee, 2007). In every phase within the CPS model, the 
divergent and convergent thinking are used in the process of problem solving. The balance between these two 
kinds of thinking in every step of the problem solving process is a critical factor for the Creative Problem 
Solving. Studies had been conducted from different definitions for creativity and problem solving. Recently, 
however, researchers have put more emphasis on studying creativity from the perspective of problem solving. 

CPS has emerged through the past several decades of work. The descriptions Osborn(1963) proposed 
have become the fundamentals of CPS. The Osbom-Pames model is the five-stage revision of Osborn’s initial 
framework that Pames(1977) developed by using an experimental study. The five stages are fact-finding, 
problem-finding, idea-finding, solution-finding, and acceptance-finding. Isaksen and Treffinger(1985) added the 
sixth stage: mess-finding. They broadened the scope of the fact-finding stage and renamed it as data-finding. 
Through the studies in the 1980s and 1990s, the process and structure of CPS have been developed in details. 
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The referred CPS models include Osborn-Pames’ model(Pames, 1977), Isaksen & Treffinger’s model(Isaksen 
& Treffinger, 1985; Treffinger et al., 2000), and Puccio, Murdock & Mance’s model(Puccio et al., 2005). 

Among these models, one of the most frequently referred conceptual model is the Isaksen and 
Treffinger’s model, a revision of Osborn-Pames’ model. Treffinger and his colleagues(2000) suggested a CPS 
model with four components, which can be classified into two major categories: a management component and 
a process component. The management component consists of 'Planning Your Approach', containing two stages 
of 'Appraising Tasks' and 'Designing Process'. This management component serves as an operating system to 
guide the application of the three process components, 'Understanding the Challenge', 'Generating Ideas', and 
'Preparing for Action'. These process components are composed of six specific stages, during which creative and 
critical thinking abilities are used in harmony. The followings are those six stages: 'Constructing Opportunities', 
'Exploring Data', 'Framing Problems', 'Generating Ideas', 'Developing Solutions', and 'Building Acceptance' 
(Treffinger et al., 2000). 

The most outstanding characteristic of CPS is the repetitive use of divergent and convergent 
thinking(Puccio et al, 2005). Divergent thinking is operated to detect and identify the problem. It facilitates 
generating various creative solutions in the process of CPS(Lee & Lee, 2007). Convergent thinking is a skill to 
detennine the solutions by focusing on many possible ideas deliberately and explicitly, and evaluating them 
with the logical, critical, analytic, and synthetic viewpoints(Kim, 2008). Therefore, for the successful 
application of CPS, it is critical to use both thinking skills appropriately and flexibly(Firertien, 1982). There are 
many techniques that can be applied within the stages of CPS to enhance the divergent and convergent thinking 
skills(Kim, 2008; Lee & Lee, 2007). Kim(2008) classified various thinking tools into the divergent and 
convergent thinking tools. For instance, brainstorming, forced connection method, morphological analysis, 
Osborn’s checklist, attribute listing, and SCAMPER are classified as the divergent thinking tools. On the other 
hand, the convergent thinking tools include hits, highlighting, reverse brainstonning, evaluation matrix, paired 
comparison analysis, ALU(Advantage, Limitation, and Unique Qualities), and PMI(Plus, Minus, and 
Interesting). 

Teaching Creativity 

Courses for creativity have been mostly focused on the CPS model with young students of primary and 
secondary school education, ranging from kindergartens to junior high schools. However, college students who 
would have to solve the complex problems in their workplace are in greater need for the creative skills. The 
studies of college courses designed to enhance creativity have focused on developing courses rather than 
investigating to develop specific models. Creativity courses for college students in Korea could be 
classified into two kinds: one is a 'teaching creativity course', with its main goal as teaching creativity itself; the 
other is a 'creativity-integrated course', in which the topics of the course are also taught while the students are 
involved in the creative problem solving process. 

Regarding the 'teaching creativity course’, Park(2004) taught creativity in his liberal-arts course of 
'Understanding and Enhancement of Creativity', and reported that the course was effective in general. 
Jeong(2003) designed a 'Developing Creativity' program for a liberal-arts course to examine the effects of a 
creativity program and the leaning styles, such as an individual and cooperative learning. The results 
demonstrated the improvement of students' creativity and effectiveness of cooperative learning. On the other 
hand, as an 'creativity-integrated course’, Baek and his colleagues(2006) implemented an 'Imaginative Design 
Engineering' course to measure and develop the creativity of the college students majoring in engineering. The 
study found that the TTCT(Torrance Tests of Creative Thinking) score of the participants has increased. 

Recently, more attentions are held to the blended learning environments for the creative problem 
solving, where both online and face-to-face learning modes are included(Graham, 2006). Lee and Lee(2007) 
developed a blended instructional model for the Creative Problem Solving and integrated an online and face-to- 
face learning environment in the formation of a 'Blended instructional model for creative problem solving'. In 
this study, they tried to enhance the Creative Problem Solving skills based on the premise that the online modes 
offer a learning environment for the divergent thinking. Students in this study would be allowed to generate 
their own ideas without any limitations. On the other hand, the offline modes provide a learning environment for 
the convergent thinking, where students’ analytic, integrated, and coherent communications are required to 
come up with some creative solutions. The researchers developed the blending principles by analyzing CPS 
models and reviewing the literature about the blended learning. 

Most previous studies on creativity have been focused on young students. Moreover, those are mostly 
about teaching creativity course in a classroom setting. Those studies have not focused much on the creativity- 
integrated courses for college students with an online support system in a blended learning environment. 
Therefore, this research aims to develop the principles and guidelines for an integrated instructional model in a 
college course using the Creative Problem Solving with an online support system. 
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Methods 

An instructional model was developed by following the three steps: discerning the general principles 
from reviewing the relevant literature; extracting theoretical components for the model by categorizing those 
general principles; and finally, developing a set of specific guidelines for each principle from the viewpoint of 
these theoretical components. The instructional model also included such components as e-Leaming contents 
and an online support system. 

The online support system was developed based on the CPS model proposed by Treffinger and his 
colleagues(2000). The system guides the CPS process and provides convergent and divergent thinking tools 
such as brainstorming, attribute list, hits, PMI, and evaluation matrix. Each stage within the system provides one 
divergent thinking tool that facilitates generating ideas and two convergent thinking tools. The first convergent 
tool is used for decision-making to select important ideas from a various ideas generated. Finally, one can 
choose the best idea with the help of the second convergent tool provided. The online support system also 
allows students to write the reflective journals. This offers them opportunities to reflect on their learning 
activities while experiencing the online CPS process. 

A college-level course was developed and implemented to examine the strengths and the weaknesses of 
the model and to suggest the improvements of the model by the fonnative research methodology. Research site 
was an undergraduate course in ‘A’ university. A total of 33 students participated, while the number of the final 
interviewees was 10. The participants' responses to the course satisfaction survey and the interviews were 
analyzed to investigate the strengths, weaknesses, and recommendations for the improvement of the model. 

Results 

Six General Design Principles 

Reviewing the relevant literature led to identify six general design principles for an integrated 
instructional model in a college course using the Creative Problem Solving with an online support system. 

Provide a blended learning environment 

The CPS model emphasizes the balance between divergent and convergent thinking in every step of the 
process to solve open-ended problems. Lee & Lee(2007) suggested that an online environment is effective for 
divergent thinking, while a classroom environment is effective for convergent thinking. 

This study, however, argued that the convergent thinking could also be effective in an online 
environment if the appropriate supportive systems were implemented. It is possible to design an online 
environment where both the divergent and convergent thinking can be exercised, and students can discuss their 
additional ideas with their team members in a classroom environment. 

Make students work on a team project 

Cooperative learning is effective for creative thinking. Choe(1998) maintained that a personal 
relationship is the crucial factor for the creative outcomes. Fie also mentioned that peers could be a new stimulus 
for producing the creative outcomes. Jeong(2003) stated that the cooperative learning can enhance not only the 
interpersonal communications and the critical thinking abilities, but also the creativity thinking skills. 

Provide an authentic task 

Creativity courses for the college students should be operated in the appropriate and relevant field 
where students usually work(Choe, 1998). In a domain-specific perspective of creativity, creativity in some 
specific fields is considered inconsistent with the ones in other fields. In short, creativity can be relatively 
independent(Han, 2000). Therefore, creativity courses should offer some authentic tasks that provide students 
with some opportunities to experience their own fields. 

Provide an online support system for the Creative Problem Solving 

Lee & Lee(2007) suggested an online support system for the divergent thinking, and Lee et al.(2007) 
identified that an online support system in the Creative Problem Solving process is effective. This study 
includes designing of the environmental supports, not only for the divergent thinking but also for the convergent 
thinking, and managing the whole processes of CPS. 

Provide appropriate guidance for the Creative Problem Solving and class management 

Since students are not familiar with CPS as their teaching-learning model, appropriate introduction of 
CPS, creative thinking tools, proper guidance, and feedback should be provided. Also, this principle includes the 
e-Leaming contents being developed to explicate the major characteristics and thinking tools of the Creative 
Problem Solving, and the tutors’ supportive roles for students. . 

Make students write reflective journals 

For a teaching creativity, many researchers stated that the internal motivations, such as devotion, 
satisfaction, and interest, are the fundamental factors to help produce creativity(Amabile, 1996). While they 
were writing journals, students can reflect on their process of the problem solving, and consequently, their 
internal motivations may be promoted 
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Theoretical Components for the Model 

Three theoretical components for the model were extracted by categorizing the general design 
principles. The first is pertinent to the environment where the learning occurs, and it includes the classroom 
environment and online environment. The second is related to the processes of instructions. There are four 
phases: a preparatory phase, a preliminary implementing phase, an implementing phase, and an evaluation phase. 
The last theoretical component would be the process components of CPS, such as 'Understanding the Challenge', 
'Generating Ideas', and 'Preparing for Action'. Figure 1 delineates three theoretical components for the 
instructional model. 




Figure 1. Three theoretical components for the instructional model 



Specific Design Guidelines 

A set of guidelines were suggested from the general design principles and theoretical components. 
Figure 2 visualizes the specific design guidelines of the integrated instructional model. 
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Figure 2. The integrated instructional model in a collegiate course for CPS with online Support system 



In the preparatory phase, the course guidance is introduced by the instructor in the classroom and 
students learn about creativity through e-Leaming contents that include the concept of creativity, the CPS 
models, and thinking tools/techniques for processes of CPS. After learning through the contents, students are 
supposed to write their online reflective journals. 

In the preliminary implementing phase, the instructor offers the guidance for a project. Students are 
divided into four teams of seven to eight members each. The students are obligated to register via online support 
system, and the instructor demonstrates how to work on a project. After a team efficiency test is administered, 
the test result is reported to members to guide individual activities for the project. 

In the implementing phase, students work on a team project with the direction of their instructor and 
tutor. In this stage, opportunities for the team discussions and instructor’s feedback are offered in the classroom, 
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and students complete each stage of CPS using the convergent and divergent thinking tools embedded in the 
online support system. Figure 3 shows the processes of CPS including the sub-processes, the contents of 
classroom instructions, and the types of thinking tools offered to the students. 



CPS process 

Understanding 
the Challenge 

• Constructing 
Opportunities 



classroom instructions online CPS support system 



* team discussions 

* instructor's monitor 



divergent ■ 
convergent 
convergent 




■ Bu Skill 3 Acceptance • team discussions 

• instructor's: monitoi 



< tutors activities > 

•pditicipating fci 
Ns/her assigned 
tecwn's B3$ 

• srasverhs or 

the quesfcyts 

• confirm na th# 
poducte o' e®ch 
piocera 

and orcwing 
feedback 




convergent 

convergent 



Figure 3. The integration of classroom instructions and an online support system based on CPS process 



In the phase 1, 'Understanding the Challenged there are three stages. In the first stage of 'Constructing 
Opportunities' students are experiencing the convergent and divergent thinking activities using the online tools, 
such as brainstorming, attribute listing, hits, PMI, and evaluation matrix. Brainstorming is one of the most 
popular divergent thinking tools through which students may generate diverse ideas. It encourages them to 
produce various types of ideas they have in mind without any limitations or hesitations. Students are asked not 
to judge the others’ ideas upon their ideas. Attribute listing is one divergent thinking tool that helps a student to 
list and analyze all the attributes in an object, or a task of a problem faced. Then, students concentrate on one or 
a couple of attributes. Finally, they change or modify those in order to come up with the alternative solutions by 
making their thinking deeper and more expanded. ‘HITs’ is a convergent tool used to reduce the number of 
proposed ideas. It allows students to put a check sign next to the more seemingly prospective ideas. It is useful 
when they need to identify the more important, interesting, and challenging ideas. PMI stands for Plus, Minus, 
and Interesting. It forces students to concentrate solely on one idea and analyze its strength and weakness, along 
with its interesting aspects. Evaluation matrix helps students to select the most appropriate solution by 
evaluating the alternative ideas that they generate based on the assigned criteria. In the second stage of 
'Exploring Data', students use the attribute listing as well as hits and PMI. In the last stage of 'Framing Problems', 
brainstorming, hits, and evaluation matrix are provided for students’ thinking processes. 

In the phase 2, 'Generating Ideas', brainstorming, HITs, and PMI can be used by students, and in the 
phase 3, 'Preparing for Action', brainstorming, hits, and PMI are offered for the stages of 'Developing Solutions' 
and 'Building Acceptance'. 

Throughout the entire stages, advices of the tutors are available for each team. They can provide the 
feedbacks on the students’ output. 

In the evaluation phase, the final output is presented and evaluated, and a peer evaluation for the 
participation level is conducted. 

General Responses to the Model 

The interviews were conducted with ten students and the participants' responses were analyzed to 
examine the strengths, weaknesses, and improvements of the model. 

Selected responses for the strengths are as follows: Providing blended learning environments, 
implementing a team project, and utilizing the CPS support system. Some responses for the weaknesses are as 
follows: Lack of integration between online and offline learning environment, improper schedule for the 
program solving processes, and complexities of the CPS system. The improvements are as follows: Seamless 
integration between online and offline environments and rescheduling the process of the CPS projects. Table 1. 
shows more details for the strengths, weaknesses, and improvements of the model. 
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Table 1. Strengths, weaknesses, and improvements of the model 





Students’ responses 


Frequency 


Strengths 


Providing the blended learning environments 


6 


Implementing a team project 


9 


Utilizing the CPS support system 


10 


Providing the e-learning modules 


3 


Providing learning resources in the LMS 


3 


Providing the tutors’ feedback 


6 


Weaknesses 


Lack of integration between online and offline learning environments 


6 


Improper schedule for the problem solving processes 


10 


Limitations of the CPS system; complexness, inflexibility, and the lack 
of explanation about terminology 


10 


The number of team members 


2 


Lack of the tutors’ feedback or help for the products of each CPS step 
and explanations for the new concepts 


4 


Improper criteria for evaluating the final products 


2 


Improvements 


Seamless integration between online and offline learning environments 


8 


Rescheduling the process of the CPS projects 


6 


Providing a variety of thinking tools 


3 


Resetting the criteria for evaluating 


2 



(Number of participants: 10) 



Responses of Respective Design Principles 

Most interviewees responded that each design principle facilitated the process of creative problem 
solving. However, some weaknesses and improvements for each design principle existed as well. Students’ 
responses for each design principles are listed in Table 2. 



Table 2. Responses of respective design principles 



Design principles 


Students’ responses 


Strengths(Frequency) 


Weaknesses(Frequency) 


Improvements(Frequency) 


Provide a blended 

learning 

environment 


It was helpful for the 
process of creative 
problem solving. (4) 


Offline activities were 
not helpful. (2) 


Instructor’s instant 
feedback on the online 
activities is needed. (1) 


Make students 
work on a team 
project 


Helpful for the process 
of creative problem 
solving. (7) 




4-5 students are 
appropriate for one team. 
(2) 


Provide an 
authentic task 


An authentic task for 
the project was helpful 
for the process of 
creative problem 
solving. (3) 




It is needed to provide a 
task for creative problem 
solving easy and interest. 
(1) 


Provide an online 
support system for 
Creative Problem 
Solving 


CPS system was 
helpful for the process 
of creative problem 
solving. (8) 


CPS system rather 
disturbed the creative 
thinking. (1) 


The improvement in 
function of each thinking 
tool is needed. (10) 


Provide an 
appropriate 
guidance for the 
Creative Problem 
Solving and the 
class management 


E-learning contents 
were helpful for the 
process of creative 
problem solving. (10) 


There were some 
difficulties to 
understand the CPS 
model in e-leaming 
contents. (7) 


The terminologies used in 
CPS model should be easy 
to understand. (5) 


Make students 
write reflective 
journals 


Writing reflective 
journals was helpful. 
(5) 


Writing reflective 
journals in every step 
was not helpful. (1) 


The numbers of writing 
reflective journals should 
be reduced. (2) 



(Number of participants: 10) 
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Conclusion 

Traditional studies on creativity have put more emphasis on the primary and secondary school students’ 
creativity. For creativity education in a collegiate level, 'teaching creativity’ as a liberal arts course has been 
provided (Jeong, 2003; Park, 2004). On the other hand, a study of a ‘creativity-integrated course’ with a subject 
matter has been explored as well(Baek et al., 2006). Recent studies about the online support system and the 
blended learning pertinent to creativity have provided us with some new perspectives on how to teach the 
creativity skills to college students. 

This study focuses on figuring out how creativity would be taught in a collegiate setting. Although 
researchers have recently paid attention to the significance of creativity, few studies discuss the integrated 
instructional models in a college level course for the Creative Problem Solving using an online support system. 
This study identified the general design principles and the specific design guidelines for an integrated 
instructional model in a college course using CPS with an online support system. The instructional model was 
developed by the following three steps: discerning the general principles from reviewing the relevant literature; 
extracting the theoretical components for the model by categorizing the general principles; and finally, 
developing a set of specific guidelines for each principle based on these theoretical components. The 
instructional model includes the e-Leaming contents and an online support system. 

Using the formative research methodology, the strengths, weaknesses, and improvements of the model 
were analyzed. The strengths included ‘providing the blended learning environments’, ‘implementing a team 
project’, and ‘utilizing the CPS support system’. Some weaknesses and improvements involved ‘making 
participants to select the appropriate thinking tools for CPS’, ‘explaining the CPS process in details’, ‘proposing 
a flexible and appropriate schedule for the problem solving’, and ‘evaluating the final product from the 
perspective of creativity. 

Through this study, major directions for the improvements of the model were suggested. Further 
studies are expected in the following areas: to improve the principles and guidelines of the model by using the 
repetitive formative research methodology and to examine the effect of the instructional model by conducting an 
empirical study. 
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Research-based Guidelines for K-12 Science Teachers to Use Blogs 

Wei Liu 

Purdue University 

Introduction 

The interest in new media for teaching and learning has highlighted the potential of innovative application in 
web-based learning environment. This interest has focused on blogs recently. A small number of educational uses of 
blogs have surfaced. This research describes how blogging could facilitate reflection and discussion in general, 
categorizes how blogs have been used in K-12 science classroom into blogging as a content delivery tool, blogging 
as a reflective tool, and blogging as a collaborative tool. At the end, guidelines are provided for K-12 science 
teachers to use blogs to form an online science education community. 

Background 

Blogs 

Weblogs were first named and described in 1997 by Jorn Barger (Blood, 2000) and then 
shortened to blogs by Peter Merholz inl999. Blogging is the act of creating and maintaining a blog. A blogger is 
someone who maintains a blog, while the blogosphere refers to the interconnected, interlinked and constantly self- 
referential community of blogs and bloggers. 

Blogs are one of hottest issues on the web, with many enterprises investing a lot of manpower and finance to 
improve functionalities of blogs which contain text, audio, and video files that archived on a Web page for easy 
access. According to blog tracking site Technorati's latest research into the explosion of personal publishing sites, 
the blogosphere is doubling in size every five and a half months. On average, a new blog is born every second. 13.7 
million bloggers are still posting 3 months after their blogs are created. 

Ease of access and use 

Free blog services are available on the Internet, such as WordPress.com and Blogger.com, make blogging 
user-friendly for both Macintosh and Windows users. It takes less than five minutes to set up a blog. With a few 
clicks, the teacher and students can configure blog sites. Once blogs are configured, the teacher and students can 
upload text, photos, audio and video files. 

The maintenance of a blog requires minimal web-page development skills; web sites exist which allow anyone 
to create a personal blog by adding text entries and images to a selected template, the developing blog being hosted 
and archived by the web site and its provider. Web sites offer this service at no cost or very low cost. 

Reflection 

Blogs provide an area where the author can develop highly personalized content which means that self- 
expression is encouraged. Preliminary research (Shoffner, 2005; Fiedler, 2003; West, Wright, & Graham, 2005) 
suggests that blogs promote reflective practice. Lin and Yuan (2006) used blog as a platform of learning reflective 
journal. In their study with 76 students, learning reflection has a positive effect on students’ learning performance. 
Correlation is significant at a significance level of 0.001 (two-tailed). About 88% of the students felt that expressing 
learning reflections on a blog was a positive experience. About 88% of the students felt that learning reflection had a 
positive influence on their learning. Students who have higher learning performance tend to take more time on 
learning reflection. 

Ray, Flocutt, and Patterson (2005) qualitatively investigated whether blogs support reflective and creative 
expression. They identified the known population a sample of K-12 teacher created, teacher centered blogs (n=138). 
Using Dhalen’s method of sampling to the identified research sample, thirty blogs were randomly selected. Sixteen 
agreed to participate in the study. Even though interviews could not be generalized to the known population or to the 
sample, the results remain useful. Twelve out of sixteen participants reported that blogging supported reflective 
practice. The depth and level of reflection, however, varied within entries and between bloggers. 

Collaboration 

Blogs allow learners to share knowledge and experiences as well as be aware of their peers’ opinions. It could 
be used as a supplementary channel to the classroom discussion. Wang and Flsu’s study (2007) provided an example 
of adopting blogs as an alternative channel to the classroom discussion. The findings of this study suggest that 
appropriately adopting blogging tool could help instructor facilitate in-depth discussion, encourage participants who 
are not comfortable to express opinions face to face to express opinions, and provide an interactive way for 
participants to debate or discuss on issues. 
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Lin and Yuan (2006) also stated that over 90% of the 76 students looked forward to receiving their 
classmates’ comments and feedback on their reflections. More than 86% of the students felt that viewing classmates’ 
learning reflection on blogs has a positive influence on their learning. 

Glogoff (2003) investigated 2 1 students geographically dispersed across North America. Blogs were 
introduced as an additional communication tool to the course’s threaded discussion forums and chat room. 
Conclusions drawn from this study are 19 out of 21 students had a positive experience with the blog, felt that they 
learned useful and relevant material, and 14 planned to join at least one new blog in the coming months. 

Students and instructors can use blogs to cultivate ideas and share them online with colleagues and others. 
Blogs provide an electronic forum for reading, writing, and collaborating with peers and others. They can give 
students and teachers a sense of connection with their peers and form an online community. 

Statement of the problem 

The use of blogs in K-12 is a relatively new phenomenon and has been documented only recently. However, 
such papers rarely discuss the blogs in K-12 science education even though many in the field are aware that the 
number of educational “bloggers ” is growing daily. This study will classify how blogs have been used in K-12 
science education into blogging as a content delivery tool, blogging as a reflective tool, and blogging as a 
collaborative tool. In doing so, this study aims to inform K-12 science teachers the appropriate way to implement 
blogs in the classroom to promote students’ reflection and collaboration. 

Method 

Data Collection 

This study is to understand how K-12 science teachers have used blogs to promote students’ reflection and 
collaboration. The research papers first had to be located. Several major databases were used including JSTOR, 
ERIC, Proquest, Web of Science, Education Full Text and EBSCO. A variety of search terms were used including 
blogging, blog, science learning, science education, reflection, collaboration, teacher blogging, and student 
blogging. This extensive study resulted in the identification of 22 papers. 

Operation definition of key terms 

Blogs are personal journals made up of chronological entries. The features of a blog include text, audio, and 
video files that are archived on a Web page for easy student access. The text files are similar to those found on Web 
discussion boards. These allow students and teachers to engage in written, two-way communication. The audio files, 
commonly referred to as podcasts (play on demand), allow students to listen to a description or explanation. And the 
video files, called vodcasts (video on demand), allow students access to a file that combines video with audio. 

Blogging is the act of creating and maintaining a blog. A blogger is someone who maintains a blog, while the 
blogosphere refers to the interconnected, interlinked and constantly self-referential community of blogs and 
bloggers. 

Criteria 

The focus of this study is to provide guidelines for K-12 science teachers to implement blogs in the classroom 
effectively. Case studies and blogs from the identified research papers were selected for potential inclusion in this 
study based on these criteria: (1) related to math and science education; (2) utilized by K-12 pre-service or in- 
service science teachers; (3) involving student contribution ; and (4) publicly available to afford access to the 
different ways the blogs were being used. Fifteen studies met these criteria. 

Review 

Blogs could support learning through promoting reflective thinking, encouraging peer support for one another, 
nurturing collaboration and relationship-building and extending learning outside classroom walls. Those features 
provide teachers with rich opportunities to use blogs in the K-12 science classroom. 

Blogging as a Content Delivery’ Tool 

Teachers’ blogs extend learning outside classroom walls through delivering course content including syllabi, 
multimedia storytelling, assignments, links, and audio and video files as necessary 

Glass(2007) used blogs into syllabus in order to provide a simple mechanism to alert everyone in the class. 

The tightly packed syllabus offered little room for the examination of classic problems and historical personages. 
Therefore, the online vehicle blog allowed the course instructor to enhance the syllabus by moving discussions on 
ancillary topics or those “we will get back to that" topics to the blog. On the blog, Dr. Glass introduced these 
interesting but tangential problems in a way that did not impact coverage of the actual syllabus. The blog allowed 
the authors to develop assignments that students could revise until correct. This was especially critical in the areas of 
information validation and citation. Working for mastery was more important than getting a one-time grade. 

Blogs could be utilized as an excellent tool where storytelling advances for both individual expressions and 
collaborative learning (Huffaker, 2005). Literacy remains paramount in learning, not only for language development. 
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but also as the foundation of all academic disciplines including science and mathematics. There are two cases as 
follows: 

J.H. House Elementary School in Conyers, Georgia uses a blog to encourage writing for third-graders. The 
teachers use blogs to spotlight select writings of children. Buckman Arts Magnet Elementary School in Portland, 
Oregon uses blogs to create a portal for all classrooms. The blog links among each teacher, showcasing photographs, 
student artwork, and classroom news. Because of the effortless deployment and maintenance issues of blogs, as well 
as the focus on content, collaboration, and documentation of completed tasks, Huffaker (2005) states that teachers 
across all academic disciplines could use blogs as an effective tool in promoting verbal literacy, one of the most 
important aspects of education. 

Blogs support communication in audio form. The audio files on blogs, commonly referred to as podcasts 
(play on demand), allow students to listen to a description or explanation. Some students may need additional 
support with the language of science and the pronunciation of science vocabulary. Others have difficulty reading and 
reviewing science texts and materials. Audio files on blogs provide a powerful tool to address these needs. 

In 2007, elementary pre-service teachers were investigated in a university science course (Redman, 2008). 
They were introduced, and listened to, a range of international science podcasts produced by science 
communicators. Podcasts appeared to offer learning opportunities of great pedagogical potential for science where 
students could be listening to engaging auditory presentations, and to be mobile while listening. Students could 
access mobile technology that allowed them to listen anywhere and anytime. 

In the study of differentiated science education, Colombo and Colombo (2007) investigated Ms. Daniels’ 
classroom. She has a master’s degree in science education and has been teaching life science for eight years. 

Through blogs, she differentiates instruction to help all students learn high-level science content and develop the 
academic language skills necessary to effectively communicate science concepts. She posts study guides that outline 
key points students should understand for projects, tests, and other assignments. She includes links to more 
advanced materials for students. 

What’s more, Ms. Daniels creates podcasts to introduce each topic within the text. She has chosen podcasts 
because they require less computer memory and download quickly. When students click on a podcast, they hear Ms. 
Daniels speaking slowly and clearly, providing time for them to locate pages and diagrams in their texts. Podcasts of 
each lesson are available to students, parents, and paraprofessionals on the class blog. 

Blogs support communication both in audio and visual form. The video files on blogs, called vodcasts 
(video on demand), allow students access to a file that combines video with audio. Teachers can combine 
PowerPoint visuals and narration to review key points of class lessons and then save these as vodcasts. They then 
upload the vodcasts to the class blog, which increases the accessibility of these lessons not only to students but also 
to parents, tutors, teacher specialists, and paraprofessionals who work with students. 

Again in Colombo and Colombo’s study (2007), Ms. Daniels uses vodcasts to support the science content 
reading lessons she conducts at the beginning of each year. When students struggle with text that is difficult for 
them, they and their parents can access these reading-strategy lessons. With one click, students are able to download 
“Using Unit Objectives” and “Learning New Vocabulary.” When students open “Using Unit Objectives,” an image 
of the first page of a unit from the textbook appears. As they see the words “Unit Objectives” being circled, they 
hear Ms. Daniels explaining, "In this text, objectives are found on the left-hand side of the first page of the new 
topic. 

The power of the vodcast was identified as partly being in the images and this visualization aspect was 
clarifying for some students. These comments refer to the vodcasting experience and the visualization aspect 

This study examines students creating science video-podcasts, but importantly, only when the science content 
was well understood and the digital resources were created by the students. In Redman’s study (2008), students 
worked on their blogs to present a vodcast to others at the end of the year. The students had been introduced to 
moviemaking in 2006 in their educational computer subject that introduced them to a range of technical skills. 
Students had opportunities to draw, photograph, research, model and create videos during the year. 

Vodcast seemed to have contributed to sound levels of engagement as is evident in the final products. 

Students re-engaged with their science understandings, personally and professionally, and affirmed their ability to 
communicate effectively. The vodcast seemed to offer a richer learning opportunity that capitalized on the positive 
aspects of visualization and learning in science, and catered for another modality of learning, now covering auditory, 
kinaesthetic and visual modes. 

Blogging as a Reflective Tool 

Blogs provide students opportunities to reflect on their own knowledge, find the words to articulate and share 
it with others, and progressively students become aware of how they learn best. Redman (2008) documented the 
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students’ reflection on conceptual understandings of lunar phases. 

Student 

I felt frustrated with the concept of this assignment in the beginning stages, that it was a “hassle” for me to be 
looking up at the moon, and that I just plain old didn’t want to wonder and ponder about the moon! ! It wasn’t that 1 
don’t like the moon; 1 just felt that I didn’t feel that it deserved my time. Fortunately I had a big change of attitude, 
as I discovered that this ongoing and hands-on style of learning was really well suited to me as a learner. 

Student 

This blogging experience has unquestionably refined my understandings of the moon and its relationship 
with the sun and the earth. By beginning with observing the moon it gave me a chance to approach science in a 
completely different light. In observing the moon as a child would I felt as though I was starting from scratch and 
found myself more will to ask questions, to myself and others, which I would not normally. 

Redman’s study (2008) also noticed that students had opportunities to draw, photograph, research, model and 
create videos during the year, and this appeared to be a situation ripe with potential for reflective learning. The 
students were able to produce a vodcast and be provided with a chance to cognitively re-engage with what they had 
learnt, remembered and forgotten. Students affirmed what aspects of the learning experience contributed to the 
success of the learning journey. 

The students found constructing the vodcast an empowering experience. The groups shared the strengths and 
weaknesses of their early conceptual understandings. Blogging seemed to have positively supported the 
reconstruction of understandings. The blogging experience appeared to have contributed to the positive perceptions 
of the vodcasting experience as a contributor to reassure students of their more refined and stable understandings. 

Student 

The knowledge that I have developed through blogging was stable and I easily slipped back into 
discussions (when working on the vodcast)’ 

Student 

I understand the lunar phases and can explain it/use models. 

Student 

I can really ‘see’ or visualize where the sun is in relation to the moon each time I look up now. 

Student 

Yes, absolutely ... remember in pictures, so doing the modeling and being ‘in’ the visual was really 
effective in helping me understand. 

Student 

It was really hard to recall in the mind, (but) images and modeling can visualize the concept. 

Blogs could be used to encourage guided discovery and knowledge construction. In a module on information 
architecture, for example, students in Glogoff s course (2005) read the professional literature from blogs and visited 
Web sites that provided tutorials and other content. After this exploration, students completed an assignment which 
synthesized what they had learned and described those concepts in a real-world. 

K-12 teachers could help students establish personal blogs to write reflections. Turnbull (2004) reports on the 
routine use of blogs in British primary schools as having positive effects, including bloggers doing better in their 
schoolwork than non-blogging peers, bloggers’ interest in their schoolwork rising, and their literacy rates and IT 
competency becoming higher than that of their non-blogging peers. There is a three-step process involved in 
blogging: scouring, filtering, and posting. The blogger visits multiple Web sites relevant to his or her topic to find 
information to which they will respond, critique or hyperlink. The blogger must then filter the results to post the 
"best of' content for readers. Through this process, bloggers are exposed to vast amounts of information on their 
given topic, even if they do not comment on everything they find. The regularity of doing this at least once a week 
creates a repetitive process where the blogger builds an evergrowing scientific knowledge based on particular topics. 

On the other hand, K-12 science teachers could use blogs to reflect on their teaching practices. Luehmann 
(2008) systematically analyzed the 316 posts Ms. Frizzle who is an urban middle school science teacher wrote over 
1 school year. Ms. Frizzle used her blog to tell stories of herself and her classroom, reflect on her practice, work 
through dilemmas, solicit feedback, and display competence, among other things. She used her blog to revisit what 
she had done and written in the past, thus multiplying the potential for reflection offered by blogs. “When I choose 
an aspect of teaching that I wish to improve, I can return again and again to that theme in my writing, which helps 
me stay focused.” 

Blogging contributed to Ms. Frizzle’s development of her vision and dispositions; led to new understandings 
of content, pedagogy, and her students; and positively affected her practice by helping her in planning and other 
decision-making processes. Ms. Frizzle’s blog not only enabled her to reflect on her experiences with the goal of 
improving her practice but also enabled her to generalize from these experiences to wrestle with important issues 
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and articulate new insights and resolutions that affected her understandings, beliefs, vision, and dispositions as an 
urban reform-minded science teacher. 

Meanwhile, scientific Blogs allow and encourage students interact with experts and others outside of the 
classroom. Because of blogs’ ability to provide thoughtful dialogue within a short time frame, they provide a 
platform within science education for discussion on current theories and recent discoveries and development to 
promote reflection. In May 2005, UniServe Science set up a blog called Blogging in Science and Science Education 
(http://scienced.blogspot.com/) to accumulate information on the use of blogs and other content aggregation tools in 
the area of science, scientific research, science education and research into science education. This type of blog is 
sometimes known as a knowledge log or klog. 

Blogging as a Collaborative Tool 

Bruner (1996) states that learners need to actively, and creatively, use tools from their culture to create their 
understandings with others. Blogs provide a platform for the interchange of information and ideas on how these 
tools can play a role in science and science education and to provide a forum for discussion of these issues. Blogging 
supports communication effectively. 

One excellent example of a blog being used in this way is Real Climate (http://www.realclimate.org/) with its 
ongoing discussion of global dimming and other issues of climate change (Placing, Ward, Peat, & Teixeira, 2005). 
Unlike many less informed blogs, this one is the work of a number of scientists working in the area of climate and 
climate change, and is their attempt to ensure the general public and the media are kept informed on key issues. 

In their own words: RealClimate is a commentary site on climate science by working climate scientists for the 
interested public and journalists. We aim to provide a quick response to developing stories and provide the context 
sometimes missing in mainstream commentary. The discussion here is restricted to scientific topics and will not get 
involved in any political or economic implications of the science. 

Blogs create communities of learning, science understandings are refined, positive identities as learners are 
created and science ideas are continually investigated and articulated. K-12 science teachers could use blogs to form 
an online community. As educators and other users post to their blogs, they are also, potentially, creating an online 
resource for their peers to refer to for guidance (Campbell, 2003). Because Blog software often includes ways to link 
users together, others can comment on a peer’s posts or even work together online to solve a shared problem. A 
science class, for instance, can give rise to an exchange of lessons learned after a scientific experiment. A discussion 
of fundamental concepts in science could help students understand the logic behind the formula. 

MacBride and Luehmann (2008) did a case study to make contribution by exploring how one high school 
mathematics teacher effectively integrated classroom blogging into the central, disciplinary work of the class. 
At the time of this study (during the 2005-06 school year), Mr. K. had been a high school math teacher for 13 
years and was using blogs for the second semester. The classroom blog he developed specifically for his 11th 
grade Pre-Calculus class (which was publicly accessible on the Internet) provided rich examples of diverse 
ways a teacher could employ classroom blogging to support student learning. Mr. K's blog for the Analytical 
Geometry unit lasted 27 days (October 25, 2005-November 20, 2005). In this time period there were 1,292 lines 
of text written on the blog; 30 posts by students; 11 posts by Mr. K; 26 comments from students; and 3 
comments from Mr. K. 

Mr. K's blog included the following key features: 

1. Mr.K. required each of his students to take a turn at writing a post that recorded what happened in class 
that day. 

2. Mr. K. also used the classroom blog to support students' reflection concerning the 
process and outcome of their learning over the course of a given unit. 

3. Chat Boxes. Noticing that a large number of student responses to posts were abbreviated comments and 
questions much like the "talk" that occurs in synchronous communication forums such as text messaging, Mr. 
K. integrated a Chat Box tool as a feature of their class blog. Mr. K. added this feature to his blog primarily to 
facilitate students' communication with each other especially regarding specific class assignments. 

4. Sharing of resources. Mr. K also used his blog to provide students with access to math specific resources such as 
assignments, reviews, online tutorial or quizzes. Students also contributed links to resources they had found useful. 

Mr. K's classroom blog 1) increases collaborative learning and nurtures a community of learners; 2) 
creates a student-centered learning environment; 3) provides a place for reflection; and 4) provides enrichment to 
the class. 

Discussion 

Science education has acknowledged a need to engage students with science, to make the learning space a 
personalized one and one that is of interest to the learner. Using blogs, students have many ways to become 
engaged and motivated in the learning experience. While science classroom blogs like Mr.K.'s are still far from 
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being common, some teachers using blogs in different instructional science contexts have found their experiences 
very rewarding and wrote about the benefits of blogging for them and their students. 

Despite the lack of rigorous empirical studies in this emerging literature especially in peer-reviewed journals, 
these review results can be valuable for K-12 teachers as they begin to identify 3 potential benefits of classroom 
blogs that may warrant consideration and further exploration. As a result, the appropriate way to implement blogs in 
the science education to promote students’ reflection and collaboration is to form an online community of practice. 
Blogging as a Content Delivery Tool 

Blogs extend learning outside classroom walls through delivering course content including text, hyperlinks, 
audio and video materials. The various forms of information promote students’ active engagement in science 
learning. 

In science education the process of visualizing has a perceptual emphasis referring to the learner’s awareness 
of an object that can be seen and or handled, but also has a focus on the imagery that is produced as a consequence 
of the experience. The mental imagery is believed to be a major strategy in cognitive tasks and aids remembering 
and understanding and is particularly valuable in science. Blogs offer all kinds of ways for students to establish 
personal and intellectual ownership of new concepts while they visualize and interact with abstract scientific 
ideas. 

Blogging as a Reflective Tool 

The potential of blogs for reflection is shared by personal journals. Blogging allows students to have a real 
audience, and thus can motivate students to do their best work. Students posting on a classroom blog know that at 
the very least their teacher and classmates will be looking at their posts; as most blogs are public spaces, this 
audience could extend even further, beyond the classroom. 

Besides, blogs have an advantage in terms that they provide easier ways to access and make connections 
across one’s previous posts. Students could use blog to revisit what they have done and written in the past, thus 
multiplying the potential for reflection offered by blogs. 

Blogging as a Collaborative Tool 

The structure of blogs has great features to support professional practice in science. Specifically, the 
progression from post to comments then a new post to more comments in many ways reflects the practice of 
professionals in the scientific community in which a hypothesis is generated, tested, analyzed (or reviewed), 
followed by a new hypothesis, and so on. 

As discussed above, forming an online science community of practice using blogs to promote reflection and 
collaboration would benefit students a lot. That community of practice refers to the process of social learning that 
occurs when people who have a common interest in some scientific subjects or problems collaborate over an 
extended period to share ideas, find solutions, and build innovations. 

Blogs would be designed to support these opportunities for growth in human cognition, through active 
sharing of ideas, to help to create meanings that are socially and culturally shared, reflected upon and 
discussed. 

Here are some guidelines to form an online science community of practice: 

1 . Decide the purpose of the blog 

Blogs can either serve as a reflective tool or collaborative tool. Teacher’s blog can deliver science content as 
well. Rules need to be set up at the beginning of the course to make sure each individual’s responsibility. For 
example, students’ blogs are for reflection. The formats could be text, audio and video. 

2. Spend time visiting other science classroom blogs 

A quick search on Google or other search engines for "blog" and the topic area should provide some 
interesting results. Finding out what challenges and successes other science teachers experienced with your content 
area could help your own learning community. 

3. Model blogging for students 

Spend some time introducing the concept of blogging, how it is done, why it is done, showing good and bad 
blogs, etc. Then, provide a set of strict rules for blogging such as frequency, length of posts, number of and staying 
on topic. This set of rules can be created together with students. 

Students should be made explicitly aware of what is not appropriate on the blog. And remember that because 
blog are Web-based, informal communication, students may be apt to use inappropriate language on their blogs. 
They might also fail to use references and citations when quoting others' work. 

4. Make blog into routine use 

Developing and maintaining an active audience will inevitably present considerable challenges for beginning 
science teachers, as it requires time to build an audience for a new blog, as well as a commitment to a minimum 
level and frequency of posting in order to provide enough attraction to regular participants. 
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5. Enrichment 

Be creative and fun! A new puzzle or problem solving game each Sunday may motivate your students into 
science learning. It does not need to be a requirement, let your students gave it a try and post some comments related 
to their attempts. 

6. Technique tips 

RSS up-to-date: RSS is a content aggregation tool. Electronic updates using RSS (Really Simple Syndication 
or Rich Site Summary) technology are often included in blogs to provide a way to aggregate forthcoming 
information on a topic or from a Web site. RSS makes a blog system more convenient. Students can actively 
subscribe interesting topics or content through RSS feeds, which in turn reduces the time for searching and querying 
information significantly. 

Social bookmarking: It is a Web based service, where shared lists of user-created Internet bookmarks are 
displayed. These shared lists can be organized and tagged with appropriate subject headings. Web based bookmarks 
that are created and tagged by librarians can be update with additional, patron-devised, tags. Del.icio.us, simpy, and 
BlogMarks, are Web sites that support social bookmarking. 

Conclusion 

The characteristics of blogs create an excellent computer mediated communication context for individual 
expressions and collaborative. Therefore, it is an effective tool in promoting science education because of the 
effortless deployment and maintenance issues of blogs, as well as the focus on content, collaboration, and 
documentation of completed tasks. Meanwhile, as a valuable e-learning tool, blogging can be used in a number of 
ways to engage students in scientific discussion, exploration, and discovery. 

However, Blogging is a time consuming issue since it involves great amount of reading, writing, responding, 
and thinking other than the required participation time in class and work to be done. Further research relating to the 
strategies and effectiveness of blogs in K-12 science education is necessary. 

The life of blogs, even that of team blogs, is always dependent on participation. Many blogs come to a dead 
end, due to either waning interest by the blogger or because they have moved on to other blogs or other forms of 
communication and participation. The challenge for the educator is to maintain interest and relevance within the 
blog. 
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Introduction 

The field of online learning faces a challenge: While we speak of online learning as a single entity, it tends 
to look very differently depending on the context of delivery. K12 and higher-ed learners, for example, might expect 
a fixed-pace, instructor-led course, whereas corporate learners see a greater mix of self-paced tutorials. Differences 
in setting, audience, techonlogy, pedagogy, and subject matter make generalizations and comparisons extremely 
challenging. Just as instructional designers often neglect the context of instruction (Tessmer, 1990; Tessmer & 
Richey, 1997; Tessmer & Wedman, 1995), practitioners and researchers of online learning rarely place enough 
emphasis on the context of their practices and models. And context changes everything. 

This is a happy problem of course, part of the growing pains of a successful infusion into established 
institutions. As online learning enters mainstream practice (Allen & Seaman, 2006, 2008; Lokken, 2009; Picciano & 
Seaman, 2007), we need more nuanced descriptions and specifications in order to guide practice and understand 
appropriate uses. This paper responds to that growing need by reviewing, synthesizing, and expanding on past 
classifications of online learning in an effort to develop an initial framework that presents key variables of the online 
learning landscape as well as a typology that can be used to classify specific instances of online learning. Our intent 
is to develop a framework to provide a more precise language for research, and help practitioners targeting quality 
assurance and program improvement. 

The Role of Context 

A need for greater attention to context should come as no surprise. History has shown that context plays an 
important role in education. From the importance of conducting a front-end analysis when designing training and 
instruction (Rossett, 1987), to the development of situated theories of learning (Brown, Collins, & Duguid, 1989; 
Lave & Wenger, 1991; Wenger, 1998), to the rise of qualitative and mixed methods of research (Dellinger, 2007; 
Lincoln & Guba, 1985; Lowenthal & Leech, 2009), academics have been placing a greater emphasis on context over 
the last few decades. This is not generally true for online learning however. Rather than focus on or even 
acknowledge the situated and contextual nature of online learning, we tend to talk about online learning as a single 
entity. The language we use shapes the way we think, just as the way we think shapes the language we use 
(Lowenthal & Wilson, in press; White & Lowenthal, 2009). This lack of specificity when talking about online 
learning is problematic because it perpetuates a myth that there is a single type of "online learning." 

An undifferentiated construct of online learning is problematic for a number of reasons, three of which are 
briefly outlined below. 

• It confounds research results related to online learning (Philips, 2005, p. 541), which is regularly criticized 
for being low quality (Amiel & Reeves, 2008; Bernard et ah, 2004; Reeves, 1995; Wray, Lowenthal, 

Bates, & Stevens, 2008). Researchers need to be as transparent and explicit as possible when researching 
anything - but especially when studying the ever-changing field of online learning. The ability to replicate 
findings is often seen as the hallmark of rigorous research and a basis for advancing knowledge (Moore & 
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Anderson, 2003). However, the lack of precision, explicitness, and agreement in the language we use to 
talk about, think about, and describe online learning makes this very difficult. 

• It confuses practitioners by glossing over key differences in practice. While we continue to hear about how 

much online learning is growing each year, comparatively speaking, we tend to hear very little about 
online attrition and the difficulties encountered by students. Research suggests that attrition in online 
learning is higher than traditional face-to-face programs (Carr, 2000; Phipps & Merisotis, 1999; Willging 
& Johnson, 2004). Attrition, retention, and student persistence are complex issues that are difficult to 
study (Hagedorn, 2005) and overall research on attrition in online learning is relatively new (Shea & 
Bidjerano, 2008, p. 346). Therefore, there is a great deal we do not know about why some students drop 
out of online courses. Part of the reason may be the broad and often inaccurate assumptions of what online 
learning entails, when in fact students find themselves in a wide vary of online experiences when taking 
courses online. 

• It likely influences how faculty and instructional designers design online courses. For instance, the overall 

popularity of asynchronous online learning environments coupled with past experience, leads faculty and 
instructional designers (especially in higher education spaces) to have a certain conception of “online 
learning.” This conception is of a primarily (if not exclusively) asynchronous learning experience which 
then influences and possibly restricts the type of instructional activities they design. 

Online learning manifests itself in many different ways and these differences need to be more consistently 
acknowledged, discussed, and valued. While dissertations might include thick descriptions of the context and type of 
online learning, journal articles (possibly due to restrictive word limits) often lack this type of rich and extremely 
important detail. We contend that when online learning is taken "out of context," we tend to have problems like 
those previously mentioned among researchers, practitioners, and students. 

Background - Different Ways of Describing Online Learning 

In the following section, we review and synthesize some common distinctions and characterizations made 
about online learning. These core concepts are later included in a list of key variables of online learning that 
researchers and practitioners alike should consider and serve as the basis of our initial proposed typology. 
Synchronous vs. Asynchronous 

Perhaps the earliest distinction made about online learning was between synchronous forms and 
asynchronous forms of online learning (Hrastinski, 2008). Simply put, synchronous learning requires the instructor 
and learner to be online at the same time to communicate and engage in instructional activities, whereas 
asynchronous learning allows instructors and learners to be online at different times, or proceed in an entirely self- 
paced mode with no instructor presence (Horton, 2006; Salmon, 2003). This distinction was especially useful with 
the rise of email, bulletin boards, and threaded discussion tools - contrasted with broadcast forms of delivery. 

The reality though is that most online learning these days - especially in K-12 and Higher Education spaces 
- is mediated by Learning Management Systems (LMSs), which now come standard with both synchronous and 
asynchronous communication tools. While asynchronous activities still dominate online learning these days (Parry, 
2009; WCET, 2009), increasingly online courses include a mix of synchronous and asynchronous activities 
(Salmon, 2003). Even when an instructor does not include any synchronous learning activities per se, he or she 
might use a chat tool to hold set office hours. Thus, it may be unproductive to characterize a course as simply being 
either synchronous or asynchronous, when in fact a mix of elements are used. 

Instructor Led vs. Learner Led 

Another commonly used way to characterize and differentiate online learning is to focus on the pacing of 
the course and the degree to which a learner can control the pacing of the course. This is often referred to as 
instructor led vs. learner led (e.g., DeRouin, Fritzsche, & Salas, 2004; Horton, 2006) or group-paced vs. self-paced 
(e.g., Brandon, 2007). Instructor led is used to describe a typical college course in which an instructor facilitates an 
online course (often self-designed and produced) with a group of students. Such a course would typically fall on a 
traditional semester schedule in a asynchronous environment. In instructor led courses, learners are expected to 
complete the course according to the instructor's predefined guidelines which typically include set deadlines. This 
structure often prevents a student from working ahead at his or her own pace (Horton, 2006). Learner led on the 
other hand is used to describe online learning environments in which the learner can proceed at his or her own pace 
(Horton, 2006). This typically takes place in workplace education rather than K-12 and postsecondary settings; 
however all levels of formal education have examples of learner led offerings. 

While this differentiation is useful, some issues remain. First, while some courses are truly instructor led, 
very few full courses are completely learner led or self-paced. The typical college self-paced course, for example, 
must be completed during a traditional semester timeframe. Further, many learner-led online learning environments 
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might have an instructor or facilitator (even if only in a tutorial role) just as many instructor led online learning 
environments might allow a learner to proceed at his or her own pace. 

Harasim's Typology 

Harasim, first in 2000 and later in 2006 - while not specifically setting out to develop a typology - offered 
an early description of the diversity that exists across the online learning landscape. She first differentiates between 
three types of online learning: 

• Online collaborative learning is a common method used by institutions of higher education; it involves 
using asynchronous, synchronous, or a combination of the two, forms of communication to bring a group of 
students and teacher together. 

• Online distance education, on the other hand, is essentially a correspondence or independent study course 
that uses technology (e.g., email) for students to access course materials and turn in assignments. This type 
of online learning is used by K-12 and post- secondary institutions. It is essentially a one-to-one or one-to- 
many model. 

• Online computer-based training “refers to the use of the web for access to online courseware or 
individualized learning modules. There is typically neither peer collaboration nor communication with an 
instructor or tutor” (Harasim, 2006, p. 63). This is often the preferred method of online learning in 
corporate and workplace spaces, adding the flexbility needed for on-the-job and just-in-time learning. 

Different Types of Education 

Citing the work of Coombs and Ahmed (1974, as cited in Harasim, 2006), Harasim places these online 
learning types within three different types of education: 

• Formal education refers to traditional education completed typically in P-16 environments for credit and 
credentialing. 

• Non-formal education refers to the type of education often described as professional development, 
completing as part of one's job duties. 

• Informal education refers to the specific lifelong learning individuals take part in throughout their life. 
Informal learning lacks the “deliberate instructional and programmatic emphases in formal and non-formal 
education” (Harasim, 2006, p. 63). 

Different Roles o f Online Learning 

Finally, whether a course is formal or non-formal, the particular role of online learning in a course can vary 
greatly. Harasim (2006) uses three categories: 

• Adjunct mode is when online learning activities are used only to supplement a course. 

• Mixed ( blended ) mode is when online activities are used as a significant part of a course. 

• Totally online mode describes courses in which the majority (if not all) of the course activities are done 
online (p. 64). 

It's no surprise that Harasim, an early pioneer in the field of online teaching and learning (see Harasim, 

1986, 1987, 1990, 1993; Harasim & Johnson, 1986; Harasim et al„ 1995), has developed a very helpful description 
of the online learning landscape. However, some continuing issues remain. For instance, even within online 
collaborative learning, very important differences may exist. Further, the rise in popularity and access of online 
conferencing applications has aided in the rise of the webinar, a form of learning activity in its own right. 
iNACOL's Classification of Online Teaching and Learning 

A recent classification of online learning has been developed by the International Association of K-12 
Online Learning or iNACOL. iNACOL's brief (Cavanaugh et al., 2009) "examines some of the aspects of online 
teaching, specifically those related to communication and interaction" (p. 4). An entire section of the brief focuses 
on characteristics of online teaching. In order to understand the range of virtual school models, they decided to 
survey and then describe each virtual K-12 school's instructional pacing, course design, delivery technology, 
instructor role, and teacher requirements (Cavanaugh et al. 2009). The additional considerations below give body to 
an emerging framework for online learning. 

• Instructional pacing. Does the virtual course follow the traditional school schedule? Other options would be: 
does not have a formal schedule, includes a required or suggested pacing chart, or does not follow any 
specific pace. 

• Course design describes whether a virtual school uses vendor produced courses as is, with modification, or 
whether they locally develop their courses (individually or by a curriculum committee). 

• Delivery technology’ refers to whether the course relies on synchronous video, synchronous classroom 
software, asynchronous software, or blended. 



163 




• Instructor role touches on whether the instructor provides instruction primarily synchronously, provides 
synchronous supplemental instruction, provides asynchronous primary instruction, leads discussion, 
evaluates non-graded activities, and/or evaluates graded activities. 

• Communication focuses on the type of teacher-to-student and student-to-student communication required. 
Options include requiring email, discussion forum, instant messaging, synchronous, and in-person 
communication between teacher to student and/or student to student. 

• Teacher requirements addresses teacher qualifications: whether or not the teacher is required to meet 
minimum certification, have minimum teaching experience, have minimum online teacher training and/or 
minimum online teaching experience. 

These additional constructs provide further specificity for online learning, and are incorporated into the 
emerging list of key variables and typology presented in the next sections. 

A Typology (Topology?) of the Online Learning Landscape 

We recognize that a typology of online learning cannot realistically take into account every possible 
variable, nor should it do so. For a typology to be useful - and therefore more than an academic exercise - it needs 
to be as comprehensive as possible while at the same time being clear and easy to use. We present in this section the 
characteristics / variables of the online learning landscape we have chosen to include in our typology. Only those 
variables likely to impact pedagogy, learning, and performance were included in the framework. 

Table 1 extends the characteristics identified by Flarasim and iNACOL and summarizes the typology. Each 
variable is then discussed below. The categories constitute a typology - but we could also use the word topology’, 
consistent with the metaphor of the online landscape! 

Table 1 



Key Characteristics of Online Learning 



Themes 


Characteristics 


Context 


Formality 




Setting 




Curriculum Fit 




Synchronous/Asynchronous 




Pacing 




% Online 




Class Size 




Development Model 




Targeted Learning 




Subject Area 


Media 


Multimedia 




3-D Virtual Worlds 


Teachers and Learners 


Instructor Role 




Cohort Group 




Student Collaboration 




Teacher Preparation 




Student Diversity 




Class Size 



Contextual Variables 

Formality. As pointed out earlier, Flarasim (2006), working from the work of Coombs and Ahmed (1974), 
distinguished between three types of education: formal education, non-formal, and informal education (see also 
Rogers, 2004). One distinction needs to be added to Flarasim’s list. We see two different types of non-formal 
learning: required and optional. An example of required non-formal online learning is when employees are required 
to complete an online training workshop on sexual harassment prevention. An example of optional non-formal 
online learning however would be when people (typically adults) complete optional "training" (e.g., attending a 
synchronous webinar or completing an asynchronous learning module online). The optional or required nature of a 
course or module can affect learner motivation and engagement. 

Setting. As mentioned earlier, online learning tends to manifest itself differently across K-12, higher 
education, and workplace education spaces. Flarasim alludes to the importance of setting when she distinguishes 
between three types of online learning (i.e., online collaborative learning, online distance education, and computer- 
based training) but in the end she does not go far enough. While differences exist even within settings (as we 
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address, e.g., under subject area), there are important differences across setting of use. For instance, in K-12 online 
learning, online faculty have to address standards, standardized testing, and even continuous communication with 
parents in ways that higher education and corporate/professional faculty do not. Similarly, corporate/professional 
spaces often benefit from the ability to “require” an employee to complete an online learning experience, which adds 
affordances and constraints (particularly costs). 

Curriculum Fit. This variable is related to the formality of a course. Some online courses and modules fit 
neatly into a larger curriculum plan, while others do not. Some smaller-scale modules, often self-paced, fit within a 
more traditional course. This link to a larger curriculum or credentialing purpose affects how students and 
instructors think of learning and instruction within the course. Prerequisites become an issue, as do the specificity of 
learning outcomes, which may be used by another course as prerequisite knowledge. 

Pacing. The pacing is another key variable that must be taken into account. There really are two parts of 
pacing that should be considered. First, does the course (or non- formal learning experience) follow some type of 
pre-defined schedule? Second, does the learner have the ability to complete the learning experience at his or her own 
pace? Regarding the first point, for years the accelerated learning movement has questioned the importance of “seat- 
time.” That is, they have argued that people (especially adult learners) can learn the same amount of content in an 
accelerated term (e.g., 8 weeks) than in a traditional semester (e.g., 15 weeks) (Wlodkowski, 2003; Wlodkowski & 
Westover, 1999; Wlodkowski, Mauldin, & Gahn, 2001). While more research needs to be conducted on this 
(especially on accelerated online courses), most can agree that there is nothing sacrosanct about a 15-week semester. 
And while seat time does not equate to learning, the amount of time available in an online learning experience 
influences the sequence and scope as well as learners’ ability to connect with each other. For instance, research on 
social presence has suggested that social presence is established early in a course, which builds the foundation for a 
Community of Inquiry to be established and subsequent learning (Garrison & Anderson, 2003; Shea, Li, & Pickett, 
2006). Shortening the duration could likely impact how learners establish a Community of Inquiry (Lowenthal & 
Lowenthal, 2009). Similarly, but for different reasons, allowing learners to progress at their own pace is likely to 
influence the types of instructional activities included. 

% Online. The increased use of course management systems (CMS) at campuses across the country and 
Web 2.0 applications is making it easier for faculty to incorporate online aspects into their face-to-face courses. 

From simply uploading their syllabus to the web to having simple discussions online, face-to-face instructors are 
using the Internet in their courses - thus blurring the line between face-to-face and online courses. Further, online 
courses and programs throughout the country commonly integrate face-to-face components - e.g., a meet and greet 
before the course begins or required face-to-face meetings - into their courses and programs. While face-to-face 
encounters like these are a positive addition, and blended learning experiences perhaps the best of both worlds 
(Shank, 2004)), they are unquestionably different than fully online courses, often leading to different learning 
activities and designs. 

Class Size. Class size is also an important variable to consider when designing or researching online 
learning. While some have argued that enrollments in online courses should be between 15-25 students, many 
online courses have anywhere from 25-70 students enrolled in them at any given time. An online course with 50+ 
students is very different from one with 15. 

Development Model. Early attempts at online teaching, at least in the higher education sector, were simply 
adaptations of classroom-based courses designed and developed by individual faculty interested in the new medium 
(Lowenthal & White, 2009). Many colleges and universities still rely on a faculty-driven design and development 
model, which Bates (1997) has characterized as the “Loan Ranger and Tonto” approach because of its heavy 
reliance on individual faculty. Flowever, as the demand for entire academic programs offered online has increased 
coupled with continued technological innovation - many institutions are realizing that the development and delivery 
of online education is an increasingly complicated process, requiring both a specialized pedagogy and a 
technological expertise possessed by few faculty (Lynch, 2005; Oblinger & Flawkins, 2006; Wray et al., 2008). 

As a result, across all sectors of the online learning landscape, there is an increased use of specialized 
individuals (e.g., instructional designers, web developers, flash developers, programmers, video experts, etc...) to 
design and develop (and sometimes even host) online course materials. These range from specific online learning 
departments within an institution, to consortiums that lease courses to other institutions, to consultants and 
publishers leasing and or selling their course materials (see Lowenthal & White). While research needs to be 
conducted on how using course materials produced by others influences faculty and student satisfaction and student 
learning (Lowenthal & Lowenthal, 2009), there is a long history of research suggesting that instructional design can 
improve the quality of course materials (for a recent example see Kidwaii, Flowell, Defrain, van Middlesworth, & 
Spielvogel, 2009). 
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Targeted Learning. For this variable we are interested in the level of learning targeted - is the course 
knowledge-based, skill-based, or targeting complex, authentic performance that integrates knowledge and skill 
together? Determining the primary kind of learning goal can be surprisingly difficult when reading research reports. 
Knowing the kind of learning being targeted can provide a context for understanding research findings and how to 
generalize them to similar cases and needs. 

Subject Area. Researchers of online learning have not adequately investigated the differences among 
specific academic disciplines (Arbaugh, 2005; Lowenthal & Lowenthal, 2009; Lowenthal, Lowenthal, & White, 
2009; Smith, 2005; Smith, Heindel, Torres-Ayala, 2008; White & Liccardi, 2006). Anderson et al. (2001) suggest 
that subject differences might exist across disciplines because of differences between "discipline related conceptions 
of the education process" (p. 13). Similarly, Lowenthal and Lowenthal (2009) argue that differences likely exist 
because faculty and students belong to different communities of practice, which are constructed and maintained in 
part upon the language its members use (Street, 1984). Online faculty in certain fields of study and practice (e.g., 
education) communicate differently than those in other fields (e.g., business). Similarly, students' expectations 
regarding appropriate forms of communication, and ultimately things like presence or even work load, are likely to 
vary across academic disciplines. While we often like to think of good teaching as good teaching (Ragan, 1999), in 
practice, good teaching always happens in a specific context with specific forms of discourse (Lowenthal & 
Lowenthal, 2009; Lowenthal, Lowenthal, & White, 2009). 

Media 

This category presently contains only two sub-items. We have kept it separate conceptually because of its 
importance for online learning and its potential for continuing development. 

Multimedia. Despite the increase in the availability of multimedia, most online courses tend to be primarily 
text-based (Parry, 2009; WCET, 2009). There has been a long history of debates about whether or not media can 
actually improve learning. This is not the place to continue this debate. But suffice it to say that people assume that 
online courses leverage multimedia when in fact, research suggests that most online courses (at least in the post- 
secondary sector) remain primarily text-based (Parry, 2009; WCET, 2009). It is important for researchers and 
practitioners alike to recognize this trend and strive to be as explicit as possible about the media usage in any online 
courses or programs they talk about. 

Virtual Worlds. Increasingly educators are exploring the potential for using 3-D virtual worlds like Second 
Life for learning - especially in online learning environments (see Bronack, Cheney, Riedl, & Tashner 2008; 
Bronack, Sanders, Cheney, Riedl, Tashner, & Matzen, 2008). The use of these virtual worlds ranges from fully 
immersed online learning experiences to blended approaches. Over time we expect immersive environments to be 
commonplace learning resources. 

Teachers and Learners 

Instructor Role. The role of the instructor in online learning environments is often described as being more 
of a “guide-on- the-side” rather than a “sage-on-the-stage” (King, 1993). However, this cliche can be taken to 
extremes; there is a fine line between being a guide on the side and being absent at critical learning junctures 
(Lowenthal & Parscal, 2008). The role of the instructor may reasonably be different across online courses. For 
instance, self-paced courses typically do not include an instructor. In contrast, many survey courses at the 
undergraduate level are fairly instructor- and lecture oriented. 

Cohort Group. Research about online learning suggests that students often feel isolated (Bischoff, 2000; 
Ludwig-Hardman & Dunlap, 2003) and alone in online learning environments (Grubb & Hines, 2000; Robinson, 
2000). Attrition rates for online programs continue to be higher than ground-based programs (Carr, 2000; Moody, 
2004; Phipps & Merisotis, 1999; Willging & Johnson, 2004). Whether an online course is part of a cohort - that is, a 
group of students completing the program together, typically in a set sequence - is likely to influence students’ 
ability to establish a learning community online as well as whether they persist and complete the program. 
Everything from group-work to project based assignments is likely to be influenced by the degree to which students 
have had past experience taking online courses together. 

Communication. From its early days, online learning has been characterized by its use of communication 
technology (e.g., asynchronous vs. synchronous). While asynchronous communication technology - primarily 
threaded discussions and email - still dominates online learning (when communication technology is used), a 
growing number of people are using synchronous communication technologies and an even increasing number are 
using a blend of the two (Chundur & Prakash, 2009). Research on asynchronous and synchronous communication 
technologies suggests that each of these technologies have their affordances and constraints (Horton, 2006). These 
affordances and constraints are likely to influence student satisfaction as well as student learning and therefore 
should be carefully thought about and described in detail when describing an online course or program of study. 
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Student Collaboration. The degree to which student collaborate with each other in an online course is an 
important variable to consider. Some instructor-led courses are very teacher-centric, with some discussion but very 
little collaboration on projects and assignments. Other courses utilize and rely on a great deal of collaborative group 
work. In our experience, for every highly collaborative course, there are many others with very low levels of 
collaboration. Rather than assume a course is collaborative because it is a group-paced course or that it is not 
collaborative because it is more self-paced, we posit that researchers of online learning need to explicitly describe 
the level of collaboration in the courses they study. 

Teacher Preparation. People tend to assume that anyone can teach. For instance, in the higher education 
sector, a terminal degree is a license to teach; that is, “knowing a subject well is sufficient training to teach it” 
(Stevens, 1988, p.64). Similar assumptions and practices can be found in workplace learning environments. And 
while K-12 sectors have a history of requiring some type of teacher training - whether a traditional teaching 
certificate or alternative/emergency license - before being able to teach, only six of the ten programs surveyed in 
Cavanaugh et al. (2009) - iNACOL’s brief cited earlier - have minimum online teacher training requirements and 
only one out of ten programs actually required teachers to have online teaching experience. As mentioned earlier, 
while good teaching is good teaching, researchers and practitioners alike have shown that teaching online is different 
than teaching in a face-to-face environment (Palloff & Pratt, 1999; Ragan, 1999; Salmon, 2003). 

A number of institutions are now requiring some form of preparation for online instructors. Prior to teaching 
online for the first time, faculty in Regis University’s College of Professional Studies must take part in a three-week 
assessment and training process online followed by an internship/mentoring process (Parscal & Florence, 2004). 
Regis is not alone. A growing number of programs are providing (and sometimes even requiring) training and 
development to teach online. For instance, faculty in the SUNY system have received exceptional training and 
development for nearly a decade (see Shea, Fredericksen, Pickett, & Pelz, 2003; Shea, Pickett, & Pelz, 2003). We 
suspect that just like in K-12 face-to-face settings (see Monk, Walberg, & Wong, 2001), the quality of the online 
teacher highly influences student learning. 

Student Diversity. Online courses are more diverse than ever before - but so are the students completing 
them. With the increase of non-traditional students (at least in the higher education sector) coupled with an ever- 
increasing number of international learners (in both the higher education and coiporate/professional sectors), more 
online courses consist of a very heterogeneous audience. The level of homogeneity in terms of prior knowledge, 
culture, and language can affect levels of course participation and learning (Parrish & VanBerschot, 2009). 

A One-Page Rating Form 

The list of variables in the previous section is not meant to be exhaustive. Rather, we outline key variables 
that need to be considered when talking about online learning. We recognize, and invite, other researchers to help us 
expand or refine the framework. Table 2 below casts the framework into a rating form for use by researchers and 
practitioners. The form is in draft mode; a polished formatted version needs to be prepared. The goal is to keep it on 
a single page for usability reasons. We believe a convenient form can have significant value. Researchers and 
consumers of research can benefit from this kind of specificity to make generalizations and interpretations easier and 
more accurate. In practice, instructors and students can benefit from this level of description through improved 
information to consumers, and for accountability and program-improvement purposes. 
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Table 2 

Draft One-Page Form for Rating or Description 



CONTEXT - Course setup, puporse, and fit 



Formality 



Formal/Credit 



Required Non-formal 



Optional Non- 
formal 



Informal 



Setting 



K-12 



Fligher Ed. 



Workplace Learning | Other [specify] 



Curriculum Fit 



Course within Credential or 
Degree 



Module Embedded within a Course 
or Credential 



Stand-Alone Module 



Synchronous 



[Specify % synchronous and asynchronous] 



Pacing 



Fixed - Standard 
Term 



Fixd - Accelerated 
Term 



Self-Paced on 



Completely Self-Paced 



% Online 



[Specify % online and on-site] 



Develpment 


Course was 


Course was 


Instructor is 


Instructor is teaching a 


Course is 


Model 


purchased 
by a vendor 


collaboratively 
designed and 
developed by a 
team or unit 


teaching a course 
designed and 
developed by 
another faculty 


course in which Web- 
based or other materials 
designed by others are 
incorporated into his/her 
own materials 


designed, 
developed, and 
taught by the 
instructor 



Targeted 

Learning 



Knowledge/memory/text 
processing 



Skills and operations 



Fligher-order/authentic 
performing 



Subject Area 



[Specify] 



MEDIA - Use and integration of multimedia and virtual worlds 



Multimedia 



Primarily Audio and 
Video 



Blended Media 



Primarily Text-based 



3-D Virtual 
World 



Fully Immersed 



Blended 



Supplemental 



No 3-D World 



TEACHERS AND LEARNERS 



Instructor Role 



Instructor - highly 
engaged/present 



Instructor - less engaged/present 



No Instructor 



Cohort Group 



Continuing Cohort or 
Established Group 



New Cohort 



Non Cohort - most students 
don’t know each other 



Communication 



Regular 

communication with 
faculty and between 
students 



Communication 
primarily with students 



Communication 
primarily with 
faculty 



Very little communication 
with faculty or students 



Student 

Collaboration 



Ongoing student collaboration 
on projects and issues that arise 



Occasional collaboration 
among students 



Student collaboration is rare 



Teacher 

Preparation 



T rained/Experience 
d Online Instructor 



Trained but First Time 
Teaching Online 



First Time Teaching 
Online 



Not Applicable (No 
Instructor) 



Student 

Diversity 



Heterogeneous [Describe in space provided] 



Homogeneous [Describe] 



Class Size 



[Specify class size or class size estimate] 
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Conclusion/Implications 

We are fully aware that trying to develop a stable and comprehensive typology of the online learning 
landscape is a challenging agenda. It may require more than an analytic exercise - students and professionals, for 
example, could report the most salient features that distinguish one product or experience from another. We thus see 
this as a work in progress. However, we also recognize that many of the current ways we talk about and think about 
online learning are inadequate in that they simply fail to take into account the complexities of online learning. 
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Introduction 

The Community of Inquiry (Col) framework was developed as a theoretical framework to support the 
practice and research of online learning. Garrison, Anderson, and Archer (2000) theorized that meaningful learning 
online takes place in a Col made of teachers and students, through the interaction of three core elements: teaching 
presence, social presence, and cognitive presence. Over the past 10 years, a great deal has been written on each of 
these three presences (Anderson, Rourke, Garrison, & Archer, 2001; Dunlap & Lowenthal, 2009; Garrison & 
Cleveland-Innes, 2005; Lowenthal & Parscal, 2008; Richardson & Swan, 2003;; Shea & Bidjerano, 2009; Shea, Li, 
Swan, & Pickett, 2003; Shea, Pickett, & Pelz, 2003; Swan, 2002, 2004; Swan & Shih, 2005). During the past few 
years, researchers have turned from just studying each element separately to studying the three elements 
simultaneously (Akyol & Garrison, 2008; Arbaugh, 2007, 2008; Arbaugh, Bangert, & Clevelan-Innes, 2009; 
Garrison, Cleveland-Innes, & Fung, 2004; Swan, Richardson, Ice, Garrison, Cleveland-Innes, & Arbaugh, 2008). 
However, despite the increased interest in studying the Col framework, researchers have not investigated how 
communities of inquiry differ across different discourse communities (Arbaugh, Bangert, & Cleveland-Innes, 2009; 
Lowenthal & Lowenthal, 2009) nor whether they manifest themselves in accelerated online programs (Lowenthal & 
Lowenthal, 2009). 

The purpose of this study was to investigate how student's perceptions of each of the elements of the Col 
framework differ across different discourse communities (specifically, business, education, computer science, and 
humanities) in accelerated (i.e., eight week long) online courses. The following paper reports the preliminary results 
of this study. 

Background 

Garrison et al. (2000; Garrison & Anderson, 2003) have argued that meaningful learning takes place in a 
Col made of teachers and students, through the interaction of three core elements: social presence, teaching 
presence, and cognitive presence. Social presence, the first element of the model, is the “ability of participants in a 
Community of Inquiry to project their personal characteristics into the community, thereby presenting themselves to 
other participants as “real people” (p. 89). Social presence has been conceptualized as having three components: 
affective expression, open communication, and group cohesion (Rourke, Anderson, Garrison, & Archer, 1999). 
Research on social presence has a long history dating back to the 1970’s (Lowenthal, 2009). 

Teaching presence, the second element in the model, is the ability of a teacher or teachers to establish and 
support social presence and eventually cognitive presence through instructional design and organization, the 
facilitation of discourse, and direct instruction (Anderson et al., 2001). While the research of Shea (2006) has 
questioned the direct instruction component of teaching presence, subsequent research by Arbaugh and Hwang 
(2006) and others (see Arbaugh et al., 2009) continues to support the inclusion of direct instruction as an important 
component of teaching presence. 

Cognitive presence, the third element in the model, is “the extent to which the participants in... a 
community of inquiry are able to construct meaning through sustained communication” (Garrison et al., 2000, p. 

89). Garrison, Anderson, and Archer (2001) conceptualized cognitive presence as developing through four cyclical 
stages — first a triggering event, then exploration, integration, and resolution — which they refer to as the practical 
inquiry model. 
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As previously mentioned, a great deal of research has been conducted on each these core elements 
individually. However, recently researchers and practitioners have begun using the entire Col framework as a guide 
to the practice and research of online learning. Researchers have used the Col to evaluate online discussions. For 
instance, Bartruff and Headley (2009) used the Col to evaluate online discussions in teacher education courses at a 
small Christian college. They found that the Col framework served as a “simple yet effective framework to describe 
the communication” (p. 800). Other researchers have combined the Col framework with the Quality Matters 
framework to enhance curriculum development and program rigor (Bogle, Cook, Day, & Swan, 2009). The Col 
framework has also been used to get a better idea of what is missing in online learning courses (Stodel, Thompson, 

& MacDonald, 2006) as well as to get a better idea of what exemplar online teachers do (Perry & Edwards, 2005). 
Others like Arbaugh (2008) have found strong empirical support for the Col to predict perceived learning and 
satisfaction. Finally, Cleveland-lnnes, Garrison, and Kinsel (2009) used the Col to help better understand the 
challenges that first-time online learners experience. 

Gaps in the Literature 

Despite the growing popularity and increased use of the Col framework, a number of gaps in the literature 
remain. We will outline a few of these gaps in the following paragraphs. 

Discipline and Discourse Differences 

Researchers of online learning have not adequately investigated the differences that might exist across 
academic disciplines (Arbaugh, 2005; Lowenthal & Lowenthal, 2009; Smith, 2005; Smith, Heindel, Torres-Ayala, 
2008; White & Liccardi, 2006). This could likely be due to the fact that too many studies on online learning tend to 
focus on a single course. While Anderson et al. (2001), early on, pointed out that differences might exist across 
disciplines due to "discipline related conceptions of the education process" (p. 13), we posit that the issue is much 
more pervasive. That is, we contend that differences exist because faculty and students belong to different 
communities of practice or what we call discourse communities. 

Communities are constructed and maintained in part upon the language its members use (Street, 1984). 
Participation in a given community requires knowing the specific language and literacy skills of that community 
(Gee, 1990, 1998, 2000; White & Lowenthal, under review). Thus acceptance within a community requires that one 
knows and employs the specific linguistic practices (reading, writing, speaking, non-verbal communication and even 
modes of thinking) of that community (Gee, 2000; Lave & Wenger, 1991; Wenger, 1998). 

Just as faculty — and budding scholars — in different content areas engage in their own unique discourse 
communities, online faculty in certain fields of study (e.g., education) also communicate differently than online 
faculty in other fields of study (e.g., business). Further, because of these differences, students' expectations 
regarding appropriate forms of communication and ultimately presence are likely to vary across academic 
disciplines. While we often like to think of good teaching as a universal concept, in practice, good teaching always 
happens in a specific context with specific forms of discourse. In fact, part of good teaching, especially at the 
graduate level, involves indoctrinating students into appropriate forms of discourse for the career of their choice. 
Therefore, researchers of the Col framework need to investigate whether each of the Col framework differs across 
discourse communities (i.e., academic disciplines). 

Course Format 

In addition to academic discipline, course format can likely influence all three presences. Course format 
can include not only the way that a course is designed and developed but also how the course is delivered. The 
design or format of an online course or program can influence how faculty and student’s develop and perceive social 
presence in general and teaching presence in particular (Anderson et al., 2001). For instance, having an online 
faculty member meet his/her students face-to-face before a course begins can affect a student’s perception of social 
and teaching presence. In addition, whether or not a student is part of a cohort also is likely to influence how such 
things as teaching presence is developed and perceived (Lowenthal & Lowenthal, 2009). Finally, the context and 
manner in which online courses are designed and developed is rarely taken into consideration. For instance, the way 
that a course is designed and developed (e.g., by an individual faculty member vs. by a team of faculty and 
instructional designers) is likely to influence the types of activities and media used (Lowenthal &White (2009) as 
well as how presence is established in the course. 

Time 

Related to course format is the issue of time. Early research on social presence suggested that things (e.g., 
task completion and group work) take longer in computer-mediated environments (Thurlow, Lengel, & Tomic, 
2004). Researchers have questioned how time affects social presence (Lin & Laffey, 2004; Tu & Corry, 2004) as 
well as the Col as a whole (Akyol & Garrison, 2008). However this research has not specifically investigated each 
of the three presences in accelerated courses. Time, similar to the issue of format, is likely to influence teaching 
presence, social presence, and cognitive presence. For instance, we contend that whether faculty and students spend 



174 




five weeks, eight weeks, or 16 weeks communicating online likely influences how each presence is developed, 
maintained, and perceived by students online. However, often these details are not adequately investigated in the 
research on each one of the presences individually or the Col collectively. 

In their recent review, Rourke and Kanuka (2009) criticized the Col framework as a whole — essentially 
because of the general lack of research conducted on the Col framework and student learning. While we agree that 
more research needs to be conducted on how a Col results (or does not result) in student learning, we also think it is 
important to get a better understanding how the previously mentioned issues — specifically, discipline and discourse 
differences, course format, and time — influence student perceptions of a Col. 

Focus of the Study 

This study was specifically designed to investigate these gaps. More specifically, the overarching goal of 
this study was to investigate whether a difference exists in each of the three elements of the Col across discourse 
communities (specifically, business, education, computer science, and humanities) in accelerated (i.e., eight week 
long) online courses. 

Methods 

This study was conducted at a private Catholic university. For the purpose of this paper, we will refer to 
this university as Catholic Western University (CWU). In the following sections, we will outline the methods used 
for this study. CWU consists of three colleges; the students for this study came from the College for Professional 
Studies (CPS) which consists of adult learners — in fact, a student must be 21 years or older to enroll in (CPS). Adult 
students at CWU complete undergraduate or graduate accelerated degree programs — which consist of five week and 
eight week courses. CPS has an estimated 12,000 students; 40% of the credits each semester are completed 
online. Participants of the study came from four different schools and disciplines — education, business, humanities, 
and computer science — within CPS. CPS divides a normal academic year into three semesters. Each semester then 
consists of three five-week and two eight-week terms. However, completely online courses at CWU are only offered 
during eight-week terms. Students completing fully online courses in the first eight-week spring term (i.e., Spring 8 
week 1, 2009) were invited to participate in this study. An email was sent out to 2303 students. The survey was 
completed 406 times (i.e., n=406). We are unable to compute a specific response rate because some of the 2303 
students took more than one online course during the given term and were asked to complete the survey once for 
each course they took. However, as a follow up to a study Lowenthal and Lowenthal (2009) conducted on 
differences in teaching presence across discourse communities, we decided to focus only solely on the responses 
from graduate students (n=191) because there is reason to believe that graduate students are more entrenched in 
specific discourse communities than undergraduates who might be new to a field of study and practice (Lowenthal 
& Lowenthal, 2009). 

A survey developed by Arbaugh and colleagues (2008) — to measure the three presences that make up the 
Col Framework — was used to collect data from students. Both Arbaugh et al. (2008) and Swan et al. (2008) have 
published information about the development and validation of the survey. After seeking permission from Arbaugh 
to use the survey, an electronic version of the survey was created. The survey was then administered to all students 
taking an online course in CPS during spring 8 week 1 2009. We closed the survey and stopped accepting 
responses a month after we administered it. 

Once the survey was closed, the data was downloaded and entered into SPSS. We coded any blanks as 99 
and any unknowns as 98. New variables were created for teaching presence, social presence, and cognitive presence 
by averaging the responses. Descriptive statistics as well as an Analysis of Variance (ANOVA) was conducted to 
investigate whether a difference exists in each of the three elements of the Col across discourse 
communities (specifically, business, education, computer science, and humanities) in accelerated (i.e., 8 week long) 
online courses. 

Results 

The demographics (see Table 1) illustrate that the sample for this study was roughly half males and 
females. Despite being an adult program, 56% of the respondents were between the ages of 21-30. Finally, over half 
of the respondents came from the school of business. At the same time, while the sample overall considers itself 
technology adept — with 76.7% rating itself as either a 7, 8, or 9 on a 1-10 scale with 10 being an expert — 35.5% of 
the students were taking their first course online. 
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Table 1 



Demographics 







Frequency 


Percent 


Gender 


Female 


92 


48.2 




Male 


99 


51.8 


Age 


21-30 


107 


56 




31-40 


55 


28.8 




41-60 


26 


13.6 




62-older 


3 


1.6 


Discipline 


Management / Business 


94 


49.2 




Computer / Info Science 


39 


20.4 




Humanities / Social Science 


16 


8.4 




Education / Counseling 


38 


19.9 



Table 2 



Technology ; SM/s aW Priory Experience Learning Online 







Frequency 


Percent 


Technology Skills 


7 


25 


13.2 




8 


63 


33.3 




9 


57 


30.2 




Total 


145 


76.7 


Previous Experience 


0 


65 


35.5 




1 


24 


13.1 




2 


17 


9.3 




3 


15 


8.2 




Total 


121 


66.1 



To investigate whether there was a difference in average score of student perceptions of teaching presence, 
social presence, and cognitive presence across different disciplines a one-way ANOVA was calculated. No 
statistically significant difference was found (see Table 3). However, while not statistically significant, there are 
some observable differences. For instance. Humanities / Social Science students had a mean teaching presence score 
of 3.63 whereas Education / Counseling had a mean teaching presence score of 2.97. At the same time, Management 
/ Business students had a mean social presence score of 3.47 whereas all three other disciplines had sub 3.0 mean 
social presence scores with Computer / Information Science with 2.82, Education / Counseling with 2.90, and 
Humanities / Social Science with 2.94. Finally regarding cognitive presence, Management / Business students rated 
had a mean cognitive presence score of 3.96 whereas Computer / Information Science had a mean cognitive 
presence score of 2.99. 

Table 3 

Descriptive Statistics 





Teaching Presence 


Social Presence 


Cognitive Presence 




Mean 


Mean 


Mean 


Management / Business 


3.35 


3.47 


3.96 


Computer / Information Science 


3.19 


2.82 


2.99 


Humanities / Social Science 


3.63 


2.94 


3.60 


Education / Counseling 


2.97 


2.90 


3.15 



Discussion 

Researchers struggle with finding the significance of results that reveal little or no statistical significance. 
The lack of statistical significance can quickly lead one to conclude that a study was unsuccessful. However, if for 
no other reason, the mean presence scores across the disciplines suggest more than anything else that students can 
and do perceive relatively respectable levels of presence in accelerated online adult programs that utilize enterprise 



176 




models of course development (when compared to the results of Swan et al., 2008). Further, while the results of this 
study were not statistically significant, this alone does not unequivocally prove that differences do not exist across 
discourse communities or that subject matter differences are not important variables. 

The College for Professional Studies at CWU has an atypical population of students in that as an adult 
accelerated college within a Catholic University, it attracts older — one could argue more mature — students than 
traditional colleges and universities. Not only does the average student work and pay for his or her own education 
but most students have experience working in the “real” world while managing multiple commitments that often 
include a family and children. Therefore, even though the sample for this study consists of a younger population 
than the typical student at CWU, it is likely that even these younger students had a full time job and responsibilities 
during the day. 

Further, and perhaps even more importantly, the College for Professional Studies at CWU employs an 
“Enterprise Model” of course development, which is essentially a centralized-standardized approach to the design, 
development, and management of online programs (Lowenthal & White, 2008). This means among other things that 
all online courses in CPS at CWU are designed and developed in a systematic process that involves using faculty as 
subject matter experts coupled with instructional designers and a host of other instructional technology professionals 
to develop high quality and standardized courses. Faculty then, once completing a three week long online training 
and assessment, end up teaching courses that are designed and developed by others and courses that they often do 
not have the ability to author (i.e., change). It is unclear, and impossible to ascertain from this study alone, the 
degree to which the enterprise environment influenced the results of this study. Further research is needed to 
compare student’s perceptions of presence in both enterprise like environments and traditional university 
environments. 

Finally, while the overall presence scores seem respectable, the social presence scores are comparatively 
very low. Given the fact that these courses are taught in an accelerated eight-week term, it is likely that the 
accelerated nature of the courses are affecting students ability to establish social presence in the first few weeks of a 
course. Future research should look at specifically comparing the results from a study and sample like this to more 
traditional samples that employ more decentralized models of online education to see if the accelerated and/or 
enterprise characteristics of this study and sample are affecting students overall perceptions of presence. 

Conclusion 

At some abstract level, “good teaching is good teaching.” Flowever, in practice, good teaching is always situated in a 
specific context that gives it meaning and helps define it. The research on online teaching and learning can be 
strengthened by explicitly documenting how teaching and learning online change (or does not change) depending on 
its context. One important component of any learning context is the academic discipline and its related ways of 
being and knowing. This study was one example of investigating the role that academic disciplines play in the online 
teaching and learning process. Even though the results were not statistically significant, they serve as a foundation 
for future studies. 

References 

Akyol, Z., & Garrison, D.R. (2008). The development of a community of inquiry over time in an online course: 
Understanding the progression and integration of social, cognitive and teaching presence. Journal of 
Asynchronous Learning Network, 12(3), (2-3). 

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer 
conferencing context. Journal of Asynchronous Learning Networks, 5(2), 1-17. 

Arbaugh, J. B. (2005). Flow much does" subject matter" matter? A study of disciplinary effects in on-line MBA 
courses. Academy of Management Learning & Education, 4(1), 57-73. 

Arbaugh, J.B. (2007). An empirical verification of the Community of Inquiry framework. Journal of Asynchronous 
Learning Network, 77(1), 73-85. 

Arbaugh, J. B. (2008). Does the community of inquiry framework predict outcomes in online MBA courses? 

International Review of Research in Open and Distance Learning, 9(3), 1-21. 

Arbaugh, J. B., Bangert, A., Cleveland-Innes, M. (2009, April). Subject matter effects and the Community of 

Inquiry (Col) framework. Paper presented at the annual meeting of the American Educational Research 
Association, San Diego CA. 

Arbaugh, J. B., Cleveland-Innes, M„ Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). 
Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry 
framework using a multi-institutional sample. Internet and Higher Education, 77(3-4), 133-136. 

Arbaugh, J. B., & Flwang, A. (2006). Does "teaching presence" exist in online MBA courses? Internet and Higher 
Education, 9(1), 9-21. 



177 




Bartraff, E. & Headley, S. (2009). Using the Community of Inquiry Model to Evaluate Online Courses in 

Teacher Education. In I. Gibson et al. (Eds.), Proceedings of Society for Information Technology and 
Teacher Education International Conference 2009 (pp. 795-800). Chesapeake, VA: AACE. 

Bogle, L., Cook, V., Day, S., & Swan, K. (2009). Blended program development: Applying quality matters and 
community of inquiry frameworks to ensure high quality design and implementation. Journal of the 
Research Center for Educational Technology’, 5(2), 51-65. 

Cleveland-Innes, M., Garrison, D.R., & Kinsel, E. (2007). Role adjustment for learners in an online community of 
inquiry: Identifying the challenges of incoming online learners. International Journal of Web-Based 
Learning and Teaching Technologies, 2(1), 1-16. 

Dunlap, J. C., & Lowenthal, P. R. (2009). Tweeting the night away: Using Twitter to enhance social presence. 
Journal of Information Systems Education, 20(2), 129-136. 

Garrison, D. R., & Anderson, T. (1999). Avoiding the industrialization of research universities: Big and little 
distance education. American Journal of Distance Education, 13(2) 48-63. 

Garrison, D. R., & Anderson, T. (2003). E-learning in the 21 st century: A framework for research and practice. 
London: Routledge/Falmer. 

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer 
conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105. 

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer 
conferencing in distance education. American Journal of Distance Education, 75(1), 3-21. 

Garrison, D. R. & Cleveland-Innes, M. (2005). Facilitating cognitive presence in online learning: Interaction is not 
enough. American Journal of Distance Education, 79(3), 133-148. 

Garrison, D. R., Cleveland-Innes, M., & Fung, T. (2004). Student role adjustment in online communities of inquiry: 
Model and instrument validation. Journal of Asynchronous Learning Networks, 8(2), 61-74. Retrieved 
August 13, 2005, from http://www.sloan-c.org/publications/jaln/v8n2/pdf/v8n2_garrison.pdf 

Gee, J. (1990). Social linguistics and literacies: Ideology’ in discourses. Brighton, England: Falmer Press. 

Gee, J. (1998). What is literacy? In V. Zamel & R. Spack (Eds.), Negotiating academic literacies: Teaching and 
learning across languages and cultures (pp. 52-59). Mahwah, NJ: Lawrence Erlbaum Associates. 

Gee, J. (2000). The new literacy studies: From ‘socially situated’ to the work of the social. In D. Barton, M. 

Hamilton & R. Ivanic (Eds.), Situated literacies: Reading and writing in context (pp. 180-196). New York: 
Routledge. . 

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge 
University Press. 

Lin, G., & Laffey, J. (2004). Effects of different activity types and time on online social presence. In G. Richards 
(Ed.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and 
Higher Education 2004 (pp. 1996-2000). Chesapeake, VA: AACE. 

Lowenthal, P. R. (2009). The evolution and influence of social presence theory on online learning. In T. T. Kidd 
(Ed.), Online education and adult learning: New frontiers for teaching practices (pp. 124-139). Hershey, 
PA: IGI Global. 

Lowenthal, A., & Lowenthal, P. R. (2009, April). Revisiting teaching presence: An analysis of teaching presence 
across discourse communities. Paper presented at the annual meeting of the American Education Research 
Association, San Diego, CA. 

Lowenthal, P. R., & Parscal, T. (2008). Teaching presence. The Learning Curve, 3(4), 1-2, 4. 

Lowenthal, P. R., & White, J. W. (2009). Enterprise model. In P. Rogers, G. Berg, J. Boettcher, C. Howard, L. 

Justice, & K. Schenk (Eds.), Encyclopedia of distance and online learning (2nd ed., pp. 932-936). Hershey, 
PA: IGI Global. 

Luke, C. (2006). Cyberpedagogy. In J. Weiss, J. Nolan, J. Hunsinger & P. Trifonas (Eds.), The international 
handbook of virtual learning environments. Netherlands: Springer. 

McKlin, T., Harmon, S.W., Evans, W., Jone, MG. (2002). cognitive presence in web-based learning: A content 
analysis of students' online discussions. American Journal of Distance Education, 75(1) 7-23. 

Perry, B., & Edwards, M. (2005, April). Exemplary online educators: Creating a Community of Inquiry. Turkish 
Online Journal of Distance Education, 6(2). Retrieved from 
http://tojde.anadolu.edu.tr/tojdel8/articles/article6.htm 

Richardson, J. C., & Swan, K. (2003). Examining social presence in online courses in relation to students' 
perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7(1), 68-88. 



178 




Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous text- 
based computer conferencing. Journal of Distance Education, 14(2). Retrieved from 
http://cade.athabascau.ca/voll 4. 2/rourke_et_al.html 

Rourke, L., & Kanuka, H. (2009). Learning in communities of inquiry: A review of the literature. Journal of 
Distance Education, 25(1), 19-48. 

Shea, P.J. (2006). A study of students' sense of community in online learning environments. Journal of 
Asynchronous Learning Network, 1 0( 1 ), 3 5 -44. 

Shea, P., & Bidjerano, T. (2009). Community of inquiry as a theoretical framework to foster “epistemic 

engagement” and “cognitive presence” in online education. Computers & Education, 52(3), 543-553. 

Shea, P. J., Fredericksen, E. E., Pickett, A. M., & Pelz, W. E. (2003). A preliminary investigation of “teaching 

presence” in the SUNY Learning Network. In J. Bourne & J. C. Moore (Eds.) Elements of Quality Online 
Education: Practice and Direction (pp. 279-312). Needham, MA: Sloan-C. 

Smith, G. (2005). Problems with e-learning we can’t ignore: One size does not fit all. In P. Kommers & G. Richards 
(Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and 
Telecommunications 2005 (pp. 1506-1511). Chesapeake, VA: AACE. 

Smith, G. G., Heindel, A.J., & Torres-Ayala, A.T. (2008). E-learning commodity or community: Disciplinary 
differences between online courses. Internet and Higher Education, 77(3-4), 152-159. 

Stodel, E. J., Thompson, T. L., & MacDonald, C. J. (2006). Learners’ perspectives on what is missing in online 
learning: Interpretations through the Community of Inquiry. The International Review of Research in 
Open and Distance learning, 7(3), 1-24. Retrieved from 
http://www.irrodl.org/index.php/irrodl/article/view/325/743 

Street, B. (1984). Literacy in theory’ and practice. Cambridge: Cambridge University Press. 

Swan, K. (2002). Immediacy, social presence, and asynchronous discussion. In J. Bourne & J. C. Moore (Eds.), 
Elements of quality online education (Vol. 3, pp. 157-172). Needham, MA: Sloan Center for Online 
Education. 

Swan, K. (2004). Learning online: Current research on issues of interface, teaching presence, and learner 
characteristics. In J. Bourne & J. C. Moore (Eds.), Elements of quality online education, into the 
mainstream (pp. 63-79). Needham, MA: Sloan Center for Online Education. 

Swan, K. P., Richardson, J. C., Ice, P., Garrison, D. R., Cleveland-lnnes, M., & Arbaugh, J. B. (2008). Validating a 
measurement tool of presence in online communities of inquiry, e-mentor, 2(24). 88-94, Retrieved from 
http ://www.e-mentor. edu.pl/_pdf/ ementor24.pdf 

Swan, K., & Shih, L. F. (2005). On the nature and development of social presence in online course discussions. 
Journal of Asynchronous Learning Networks, 9(3), 115-136. 

Thurlow, C., Lengel, L., & Tomic, A. (2004). Computer mediated communication: Social interaction and the 
Internet. Thousand Oaks, CA: Sage. 

Tu, C.-FL, & Corry, M. (2004). Online discussion durations impact online social presence In C. C. et. al. (Ed.), 

Proceedings of Society for Information Technology’ and Teacher Education International Conference 2004 
(pp. 3073-3077). Chesapeake, VA: AACE. 

Wenger, E. (1998). Communities of practice. Learning, meaning, and identity. Cambridge, UK: Cambridge 
University Press. 

White, J. W., & Lowenthal, P.R. (under review). A tacit “code of power”: Minority students and their relationship to 
academic discourse and the development of academic identity. Submitted to Review of Higher Education. 

White, S. A., & Liccardi, I. (2006) Flarnessing insight into disciplinary differences to refine e-learning design. 

In 36th Annual ASEE/IEEE Frontiers in Education, October 28th-3 1st October 2006, San Diego CA. 



179 




A Study on the Relations of Spatial Ability, 
Mental Model Constructs, and Learning Effectiveness 



Yue Ma 

North Carolina A&T State University 



Abstract 

This research study explored how spatial visualization and spatial relations impacted building mental 
models and affected learning performance on college students engaged in learning a statistical concept. The research 
design incorporated a Pearson correlation, factor analysis and multiple linear regression that explained factors of 
which will affect learner’s performance. The purpose was to construct a relation between spatial ability factors and 
the mental model construct, and to assess the underlying construct. 

Introduction 

It has always been a controversial topic in terms of whether computer animations facilitate learning and 
produce positive learning effectiveness (Hegarty, Kriz, & Cate, 2003; Tversky & Morrison, 2002). Some researchers 
commented that animations might be too complicated or too fast to be correctly perceived. Since animations are 
usually understood as a sequence of steps, it might be more effective if visualization could help people understand 
the animated events (Tversky et al., 2007). In relation with the sUucture of visualizations, a lot of systems, for 
example, biological, physical or conceptual such as the statistics concept have spatial and temporal sttuctures. 
Visualization can clearly describe the parts and their relations, therefore, enhance learning (Tversky et al., 2007). 

Other recent research studies on visualization and animation concentrated on how spatial visualization can 
help build mental models. Bertel, Vrachliotis, and Freksa (2007) stated that the power of cognitive design tasks 
resided in the combination of mental models, visual perception and visual mental imagery. Hegarty, Keehner, 

Cohen, Montello, and Lippa (2007) reported that spatial abilities, including visualization, spatial rotation, and spatial 
orientation helped to build and kept a mental model of knowledge sUucture, therefore enhanced learning. They 
confirmed that spatial ability was important for perceiving computer visualizations. Hegarty et al. also stated, “We 
currently know relatively little about how learners interact with visualizations and how to best design these 
visualizations to aid in the acquisition of spatial mental models or spatial skills” (p. 293). 

It is apparent that there is a need to further investigate the relations among spatial visualization and 
relations and learning effectiveness. Some researchers commented that animations can help learners build mental 
models of the concepts presented (Narayanan & Hegarty, 2002). Narayanan and Hegarty maintained that there was 
scant research based on the mental models of how people learn from graphical representations such as visual 
dynamic displays. Empirical research results have demonstrated that the effectiveness of comprehending visual 
dynamic presentations were more associated with comprehension processes than with the interactivity of the 
presentation media (Narayanan & Hegarty, 2002). 

In light of computer 3D virtual learning environment, the following research questions were proposed: 

• Are spatial visualization and relations correlated? 

• What is the dimension underlying the six mental model construct? 

• How accurately can knowledge transfer be predicted from a linear combination of comprehension 
measures? 



Literature Review 

The present study adopted cognitive learning theory and Narayanan and Hegarty’ s mental model. Cognitive 
learning theory believes that in memory there exists an organized structure, or schema, which constitutes the 
knowledge of the world. Schema is highly abstract and could be generalized. Schema could be changed through 
either instruction or personal experience and it is very dynamic. Schema can provide a context to understanding new 
knowledge. Schema can also provide a structure to hold the new knowledge (Winn, 2004). 

Spatial capability and spatial knowledge are important to life since everything existing must have a solid 
spatial location. In order to understand human cognitive organization, we have to study how people navigate and 
mentally represent and manipulate spatial information. The internal comprehending of the spatial world would drive 
the machineries of constructing mental tools to understand the spatiality (Newcombe & Huttenlocher, 2000). 

Newcombe and Huttenlocher (2000) believed that spatial capability is a very unique aspect in cognitive and 
computational levels of analysis. Spatial representations constitute spatial factors that form intelligence and 
reasoning. There are three major theories that entail the concept of spatial development. One is Piaget’s approach of 
spatial development. One is Nativist’s approach and another one is Vygotskyan's views. 
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Piaget states that children do not have knowledge of space when they are born. They do not occupy the 
capability of identifying space. Gradually they begin to foster their own ability of treating objects around them. 
Piaget argues that the initial understanding of spatial concept is formed by any objects that are either reachable or 
unreachable. Piaget describes the spatial development as different stages. He states that children see objects as 
closed to one another, not close to one another at their initial stage of understanding space. More mature 
comprehension comes in when they are nine or ten years old. He mentions that this stage is called as projective and 
Euclidean space. He defines projective space as the coding of the orders between different lines, and Euclidean 
space as the coding of objects metrically. Piaget indicates that adults are more accurate in spatial tasks (Newcombe 
& Huttenlocher, 2000). 

Nativist believes that spatial understanding may be available to infants naturally. There are three arguments 
and evidence that support Nativist’ ideas. The first argument is that spatial analysis capability exists early. Others 
believe that spatial ability develops very slowly and it is possible to develop spatial relations. The literature on blind 
children suggested that with given input, their spatial abilities increase rather than they possess these abilities at an 
early time (Newcombe & Huttenlocher, 2000). 

Second, Nativist’ s ideas argue that understanding space is related to geometric module development. 

Young children depend on recognize relative position of a landmark to provide them cues. Research studies have 
found that young children used those permanent objects to search their objects after disorientated. Third, there is an 
argument that maturation of the brain areas assumes a unidirectional casual path that is currently unjustified. 
Nativists believe that environmental feedback assist to form spatial relation rather than a specific object work as a 
trigger for spatial ability. Nativism does not provide any focus on the strength of cognitive competence and any 
input on developmental change (Newcombe & Huttenlocher, 2000). 

Vygotsky’s ideas of spatial competence come from children's ability to understand the world guided by 
adults. These guided participations are important part of the development of certain spatial skills. The second 
thought is that cognitive effort is very adaptable with specific situations, which involved spatial tasks. For example, 
a security guard may provide a better direction than a librarian because of the situation specification. Third, 
Vygotskyan investigators are very interested in dealing with maps or diagrams. Vygotskyan researchers are also 
interested in learning the interaction of individuals with their cultural environment, from which a representational 
system is acquired. Children interact with their environment as they structure their social interactions. Vygotskyan 
ideas overly emphasize on the role of the social environment in developing individual thinking. Vygotskyan ideas of 
“zone of proximal development” are similar as Piagetian ideas of cognitive readiness. Thus, the complete theory 
should involve a developing child, a skilled adult and a cultural medium, while avoiding the idea of cultural input as 
all of the development (Newcombe& Huttenlocher, 2000). 

As more microcomputers have been used in educational programs, students have the option of make 
predictions and assess hypotheses by interacting with computational simulations (Hegarty, 2004). The assumptions 
are that there are no clear evidences to show dynamic displays produce advantage over static media. One argument 
is that people can mentally animate static diagrams together to form an animation. Another argument is that it is 
expensive to produce dynamic media and producing media that cannot facilitate visualizations internally may not 
help learning (Hegarty, 2004). 

One research study demonstrates that dynamic visual displays hold the advantage of exceeding static visual 
displays in knowledge transfer (Ma, 2008). In this research study, two types of visual presentations and their effects 
on college graduate students’ cognitive load performance and knowledge transfer has been examined. The focus was 
on whether animation and static visual displays could enhance learning outcome. Graduate students consisting of 
112 individuals participated in the study. Results indicated that students in the animation group had significantly 
higher knowledge transfer score than those in the static visual group. Students in the animation group also had 
higher mental efficiency scores than those in the static visual group. Post hoc data analysis results again confirmed 
the original result that the computer animation group out-performed the static visual group on the knowledge 
transfer test (Ma, 2008). Spatial orientation and visualization were used as two individual variables. The results 
reviews that animation is more associated with visualization with post hoc analysis result indicating a significant 
effect. Spatial orientation and visualization should be used as two individual variables to measure spatial ability 
(Ma, 2008). 

Hegarty (1992) stated that when a machinery system has been presented, people started the mental 
animation. First they decomposed the representation of the system into smaller units. They then animated these 
smaller units in a sequence that constituting a casual relation. Making a causality that applied to the system was a 
preference that people tended to do. Hegarty (1992) mentioned that the mental animation model should include the 
encoding and comparison processes. 
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Hegarty, Kriz, and Cate (2003) stated that understanding a dynamic system involved several types of 
knowledge. First, it required the understanding of basic units of the system. Then, it involved how each component 
of the system moves and the chain effect. The understanding of the system will enable the understanding of the 
structure and behavior of the function, therefore, mental animation ability was highly correlated with spatial ability. 

The research study also stated that when people read a text depicting the machine motion, they tended to 
construct mental models of the machines. Hegarty, Kriz, and Cate (2003) also stated that individuals with high 
spatial capabilities usually understood animations more accurately and these individuals may have more working 
memory to integrate different components of motion. 

Narayanan and Hegarty proposed a mental model of how people comprehended multimodal presentations. 
Its purpose was to describe how the mental systems worked and how to use this model when designing applications 
using multimodal presentations. Narayanan and Hegarty stated that the model regarded comprehension as a series of 
steps in which the user integrated his or her prior knowledge into the concept domain and constructed a new mental 
model. They also said that this model was highly coordinated with spatial visualization processes, which helped to 
comprehend graphic displays (Narayanana & Hegarty, 2002). This mental model was used to study the relation 
among spatial cognition, mental model construct, and learning effectiveness. 

The constructs of this model are: (1) user constructs a mental model of dynamic graphic system by first 
decomposing its simple components, (2) representationally connecting prior knowledge, (3) mentally encoding both 
spatial and semantic relations between each concept components, (4) hypothesizing causality, (5) mentally 
animating the model and inferences further, resulting in (6) comprehension of the basic information, for example, 
involved in the logic or physics (Narayanana & Hegarty, 2002). 

Methodology 

Participants 

In this research study, 27 southeastern state university students participated in the workshop and most of 
them are undergraduate students. These undergraduate students were introduced to SecondLife 3D virtual 
environment. SecondLife is a 3D virtual environment created by its residents. The company was founded in 2003 
and its residents are engaged in education, music, technology, fashion, real estate, and business world that are 
digitalized. SecondLife is collaborating with higher education settings to promote the benefits of using virtual world. 
Data collection was performed in Spring 2009 and Fall 2009. 

Procedures 

A consent form was distributed by the principal investigator of this research study and also the guidelines 
of the study were informed. Statements were delivered to the students that they have the right to choose to 
participate in the study or withdraw at any time. Once the students finished filling out the consent form, the 
researcher started the workshop. SecondLife was introduced and the researcher started to log into the system and 
locate the avatar. Then the statistical concept of population and sample were introduced to students using 
SecondLife environment. 

The researcher divided the instruction into 6 parts based on the mental model. First, the statistical concept 
was decomposed to simple components and elaborated in SecondLife environment. Secondly, students were 
encouraged to representationally connecting prior knowledge using the 3D virtual world. Thirdly, students were 
encouraged to mentally encoding the concept components. Fourthly, researcher started to provide more examples to 
hypothesize causality. Fifthly, researcher instructed students to mentally animate the concepts further to infer to 
more cases. Last, researcher instructed students to mentally comprehend the concepts. 

After the presentation, a package of survey forms was given to students. A paper copy of spatial 
visualization and spatial relations tests produced by ETS were distributed to students to gather their spatial 
visualization and relations scores. A paper copy of knowledge transfer test and a survey to measure the effectiveness 
of each construct of the mental model was distributed to students. 

Data Analysis 

• Research question 1 : were spatial visualization and relations correlated? 

The Pearson correlation coefficient was .496, and p < .05. If correlation coefficient was around .50, then it 
was interpreted as large coefficient. The spatial orientation and visualization were correlated. See figure 1 . 
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Figure 1 . The correlation graph between spatial orientation and visualization. 

• Research question 2: What is the dimension underlying the six mental model construct? 

Two assumptions were performed before the factor analysis. Assumption 1. The measured variables were 
linearly related to the factors plus errors. To determine whether the variables were linearly related, the scatterplots 

2 

for pairs of items were performed and this assumption was met. Assumption 2. The % test for the maximum 

2 

likelihood solution assumed that the measured variables were multivariately normally distributed. The % test for 
the maximum likelihood was performed and yielded non-significant result with p > .05. The test assumed that the 
model did not constitute a multidimension. 

Factor analysis was performed to structure the underlying dimension of the six mental constructs. The goal 
of running factor analysis was to reduce the overlapping measures to a smaller set of variables. The objectives were 
(1) measuring the six indicators and decide which one(s) was a poor indicator of the construct; (2) the item scales to 
measure each construct. Factor extraction and factor rotation were performed. 

Factor extraction was performed. Eigenvalues were observed for counting the amount of variance of 
variables accounted for by a factor. Eigenvalues were the indicators to decide how many factors should be used in 
the analysis. One criterion was to keep factors that have eigenvalues greater than 1. The other criterion was to 
examine the scree plot and retain all factors with eigenvalues in the sharp decline part of the plot. 

Table 1. Initial Analysis from Factor Extraction 



Component 


Initial Eigenvalues 


Total 


% of Variance 


Cumulative % 


1 


3.966 


66.103 


66.103 


2 


.678 


11.297 


77.399 


3 


.576 


9.602 


87.001 


4 


.344 


5.735 


92.736 


5 


.277 


4.619 


97.354 


6 


.159 


2.646 


100.000 



The eigenvalue associated with the first factor, decomposing, was 3.966 and the percent of total variance 
accounted for by the first factor was 66.103%. There were two rules that we can base to do the factor rotation 
analysis. One was to base on eigenvalue greater than 1. Second was to base on scree plot. Based on the scree plot, 
the factors that were to retain were the ones that in the sharp slope of the plot before the eigenvalues start to level 
off. Factor rotation was performed based on eigenvalue greater than 1. See figure 2 for scree plot. 






Scree Plot 




Figure 2. Scree plot of the eigenvalues. 

The next stage was to rotate a one-factor solution. The decision of the underlying structure came from the 
rotated solution. The factor matrix and rotated factor matrix were shown in table 2 and 3. The matrix showed the 
factor loadings, which were the correlations between each variable. 

Table 2. Factor Matrix 





Factor 


1 


2 


Decomposion 


.709 


.079 


Representation 


.111 


.363 


Encoding 


.764 


-.141 


Hypothesizing 


.793 


-.434 


Mental_animation 


.761 


-.071 


Comprehension 


.871 


.236 



The dimensionality of the 6 items was analyzed using maximum likelihood factor analysis. The scree plot 
and eigenvalue greater than 1 both indicated that this model was a unidimensional construct. The factor matrix 
reviewed that all factors contributed to the model construct. Rotated factor matrix reviewed that there were 
underlying factors. 

The first dimension of this model was constituted by representation, mental animation, and comprehension. 
The first step of this comprehension procedure was that user connected with prior knowledge; then user mental 
animated the procedure; and then user comprehended new knowledge. 

The second dimension of this model was constituted by encoding, hypothesizing, and mental animation. 
The first step of this comprehension procedure was that user encoded both spatial and semantic relations; after that, 
user assumed causality; and then mental animation happened for further comprehension. This construct can be 
named as mental comprehension (See table 3). 

Table 3. Rotated Factor Matrix 





Factor 


1 


2 


Decomposion 


.569 


.430 


Representation 


.813 


.271 


Encoding 


.458 


.628 


Hypothesizing 


.277 


.861 


Mental_animation 


.503 


.575 


Comprehension 


.794 


.427 



• Research question 3: How accurately can knowledge transfer be predicted from a linear 
combination of comprehension measures? 

To answer research question 3, a linear multiple regression test was run. The regression equation with all 
six comprehension predictors was not significantly related to comprehension index, R~ = .21, F (6,20) = .892, p 
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=.52. the ( J > s, as labeled on the output, were the weights associated with the regression equation. The standardized 
equation was as follows: 

Knowledge transfer = -.34 decomposing + .62 representation + .18 encoding + .06 hypothesizing + .06 mental 
animation -.48 comprehension. 

In this model, six comprehension predictors were used and the factor that was loaded with the most weights 
was representation, which involved the procedure of representationally connecting with prior knowledge. This 
procedure loaded .62 standardized weights on the equation. The next weight index was encoding, which loaded .18 
standardized weights. Encoding was the procedure that learners mentally interpreted the spatial and semantic 
relations between each concept. For this data collection group, decomposing and comprehension were two factors 
that hindered learning procedure. Decomposing involved the procedure of breaking the concept into small 
components and comprehension involved the procedure of understanding the basic rules of how this complex 
procedure operated. For this group set, these two procedures lowered the effectiveness of the model (See Table 4). 

Table 4. Coefficients 



Model 


Unstandardized Coefficients 


Standardized Coefficients 


Constant 


51.28 




Decomposing 


-9.14 


-.34 


Representation 


20.14 


.62 


Encoding 


5.01 


.18 


Hypothesizing 


1.95 


.06 


Mental_Animation 


1.70 


.06 


Comprehension 


-14.18 


-.48 



Discussion 

The objectives of this study were to test the correlation between two spatial concepts. One was spatial 
orientation and the other one was visualization. The study result reviewed that visualization and spatial orientation 
were correlated. Learning a 3D virtual world concept, spatial orientation and visualization would contribute to the 
learning effectiveness. 

The second objective was to measure the dimensionality of the six factor model. The factor matrix result 
reviewed that this model was a unidimensional construct. The rotated factor matrix reviewed that underlying factors 
existed in this model. The first dimension of this model was constituted by representation, mental animation, and 
comprehension. The second dimension of this model was constituted by encoding, hypothesizing, and mental 
animation. Since the data set in the study only had 27 subjects. Further data collection was needed to study the 
dimensionality of the model. 

The multiple linear regression was performed and the result reviewed that representation was contributing 
most to knowledge transfer. Representation factor was associated with connecting prior knowledge, which 
demonstrated that connecting with prior knowledge was the important step for building mental model construct and 
achieve learning effectiveness. 
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Abstract 

The purpose of this study was to investigate the relative effectiveness of positive interdependence 
and group processing on student achievement, interaction, and attitude in online cooperative learning. All 
of the participants, 144 college students enrolled in one of three different courses, received initial general 
instruction about teamwork skills and cooperative learning at the start of the study. Participants were then 
randomly assigned to one of three treatment groups: positive interdependence, group processing, and no 
structure. The Positive Interdependence groups received subsequent positive interdependence skills 
training which were then utilized in their instructional activities. The Group Processing groups received 
subsequent group processing skills training for use in their instructional activities. The No Structure 
groups received no additional instructional treatment beyond the initial basic teamwork and cooperative 
learning training. 

Results indicated that there were significant differences among students in the Positive 
Interdependence, Group Processing, and No Structure groups with respect to their achievement scores and 
interactions. Participants in the Positive Interdependence groups had significantly higher achievement 
scores than participants in either the Group Processing groups or the No Structure groups. In addition, 
participants in the Positive Interdependence groups and the Group Processing groups interacted with each 
other to a greater extent than those in the No Structure groups. Regarding student attitude towards the 
experiences of cooperative learning (i.e. participation, communication resources, and online activities), 
there was no significant difference among any of the three groups. The overall results of this study suggest 
that instructors would be advised to incorporate positive interdependence strategies in their online courses 
to help students actively contribute to their online group activities. In addition, instructors are 
recommended to inform groups of the individual progress of each member’s activities periodically by 
employing group processing strategies. 

Keywords: Positive interdependence, Group processing, Online cooperative learning 

Introduction 

Unbalanced participation, or the existence of students who escape from responsibility in group activities 
(free-loaders /free-riders) has been identified by some researchers as a significant potential problem in online 
cooperative learning environments(McWhaw, Schnackenberg, Sclater, & Abrami, 2003; Slavin, 1995). This 
problem can lead to students’ unwillingness to participate in online cooperative learning activities and to negative 
influences on group productivity. The problem can be addressed effectively by 1 ) investigating the essential 
components of online cooperative learning and 2) utilizing computer and telecommunication technology which 
enables meaningful asynchronous student communication in online learning environments(Buchs, Butera, & Mugny, 
2004; Gunawardena, Lowe, & Anderson, 1997; Hancock, 2004; Nachmias, Mioduser, Oren, & Ram, 2000). 
Literature review showed positive interdependence and group processing activities as two essential components of 
online cooperative learning(Brewer & Klein, 2006; Johnson, Johnson, & Smith, 2007; McCafferty, Jacobs, & 

Iddings, 2006). Accordingly, research on the relative effectiveness of positive interdependence and group processing 
on student achievement, interaction, and attitude in online cooperative learning environments should prove to be 
useful in helping educators develop more effective instructional practices. 

Research questions addressed by this study were: 

1 . Are there any differences in the relative effectiveness between positive interdependence, group 
processing, and no structure on student achievement in online cooperative learning environments? 

2. Are there any differences in the relative effectiveness between positive interdependence, group 
processing, and no structure on student interaction in online cooperative learning environments? 
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3. Are there any differences in the relative effectiveness between positive interdependence, group 
processing, and no structure on student attitude in online cooperative learning environments? 



Table 1. Contrast Coefficients for Each Group 



Group 


Contrast 


1 2 


3 


1 


1 1 


-2 


2 


1 -1 


0 



To test these specific research questions, this study employed the method of planned orthogonal contrasts. 
Planned orthogonal contrasts are used when the contrasts are mathematically independent from each other, and they 
enable a researcher to test specific research questions that are meaningful to the researcher (Cohen, 2003). The 
above research questions are focused on whether the effects of positive interdependence (Group 1) and group 
processing (Group 2) are different from the effect of no structure (Group 3) and whether there is a relative effect 
between positive interdependence (Group 1) and group processing (Group 2). These focused research questions 
were tested with planned orthogonal contrasts in this study. Table 1 shows contrast coefficients for each group in 
this study. 

Method 

Participants and Description of the Research Design 

This study examined the relative effectiveness of positive interdependence and group processing on the 
dependent variables: student achievement, interaction, and attitude. The activities incorporated in the various 
cooperative learning groups (positive interdependence and group processing, and no structured cooperative learning 
method) formed the independent variable. An existing course in each of three different universities in South Korea 
was selected as a means of examining these activities in a broad, real-world setting. Each course incorporated all 
three conditions using independent groups of students in the respective activities. 

Prior to the start of the classes, the researcher conducted two workshops for the three instructors focused on 
teaching cooperative learning and how to incorporate positive interdependence and group processing activities. Each 
instructor was provided with general training on conducting online activities and then specific training for the 
individual groups: Group 1 (Positive Interdependence) utilized the Instructors’ Guideline for Teaching Students 
Positive Interdependence (Johnson, Johnson, & Holubec, 1994), Group 2 (Group Processing) utilized the 
Instructors’ Guideline for Teaching Students Group Processing (Johnson & Johnson, 1993; Johnson et ah, 1994); 
Group 3 (No Structure) had no associated additional resources. The total 144 participating students were randomly 
assigned to three treatment conditions (144 = 48 X 3). To facilitate active online discussion, the 48 members of each 
treatment group were then randomly assigned to 12 small discussion groups with four students in each group. All of 
the 144 participants received initial general instruction about teamwork skills and cooperative learning at the start of 
the study. Participants then received instruction specific to the activities relative to their assigned treatment group: 
positive interdependence, group processing, and no structure. The Positive Interdependence groups received 
subsequent positive interdependence skills training using the Students’ Guideline for Learning Positive 
Interdependence (Johnson et ah, 1994) which they subsequently utilized in their instructional activities. The Group 
Processing groups received subsequent group processing skills training for use in their instructional activities using 
the Students’ Guideline for Learning Students Group Processing (Johnson & Johnson, 1993; Johnson et ah, 1994) 
and the Processing Interim Report (Weekly), which was developed by Johnson, Johnson, and Holubec (1994). The 
No Structure groups received no additional instructional treatment beyond the initial basic teamwork and 
cooperative learning training. 

Instructional Conditions 

The researcher and the instructors prepared online discussion rooms in the existing respective course 
management systems in each of the three universities. The course content for online student activities was specified 
by each instructor in order for the students to complete their group reports in their respective courses. Accordingly, 
in University A, each group discussed how to develop an instructional tool, WebQuest, as a group product. They 
then developed their own WebQuest using Web-editor software like Dreamweaver and FrontPage. In University B, 
members of each group played the role of Human Resource Development (HRD) specialists and discussed how to 
develop a 6-hour instructional program. In University C, each group discussed the analysis of a class and the 
processes involved in employing learning theories to teach it. 

During the ensuing interactive instruction period, students exchanged asynchronous written 
communications in their individual groups to discuss their learning project with other members. The online 
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discussion period lasted for 3 weeks. Throughout this discussion period Group 1 (Positive Interdependence) 
members could access the Students’ Guideline for Learning Positive Interdependence (Johnson et al., 1994) in each 
online discussion room and follow it during the activity. They also rotated their complementary and interconnected 
roles every week. The instructors also helped Group 1 members achieve positive interdependence by using 
techniques presented in the Instructors’ Guideline for Teaching Students Positive Interdependence. Group 2 (Group 
Processing) members followed the Students’ Guideline for Learning Students Group Processing. Each member of 
Group 2 filled out the Processing Interim Report forms every week and submitted them in their online discussion 
rooms on the 2 nd Sunday, 3 rd Sunday and 4 th Sunday. The instructors helped Group 2 members enhance their group 
processing by using the Instructors’ Guideline for Teaching Students Group Processing. The instructors only 
answered Group 3 (No Structure) members’ questions and did not provide any extra directions for online discussion 
or specific methods of their cooperative learning. On the last day of the 3 rd week, each small discussion group in all 
the three conditions submitted the group report to be evaluated as the group’s final assignment product. 

Instmments 

The Rubric for Assessing Learner Writing was adapted from the instalment used to measure five criteria 
for university writing (Barlow et al., 2006). In this study, the rubric was used to examine the differences in student 
achievement (writing a group report) as the results of positive interdependence, group processing, and no structure. 
The rubric contained five criteria: purpose, evidence-based reasoning, management of flow, audience awareness, 
and language control to evaluate writing in higher education. This study used only overall scores which were simply 
summarized from five criteria to evaluate the quality of each group’s product. Instructor assigned an appropriate 
grade to the group reports from A+ to F (A+ (100), AO (95), A- (90), B+ (85), B0 (80), B-(75), C+ (70), CO (65), C- 
(60), D+ (55), DO (50), D-(45), and F (0)). However, the final grade was based on both the knowledge of the course 
content and group participation (the group product (60%) + group participation (40%)). Each instructor evaluated 
how individual members participate in their online discussion well. Students had to post meaningful online 
messages at least two times every week. With this criterion, instructor measured the levels of participation in online 
discussion. Instructor also assigned an appropriate grade to the group participation from A to D (A (100), B (90), C 
(80), and D (70)). 

To analyze student interactions, the researcher counted the total number of messages which were posted by 
each group member in each condition. This study also used the adaptation of the Interaction Analysis Model 
(Gunawardena, Lowe, & Anderson, 1997) to analyze characteristics of online interactions. The Interaction Analysis 
Model was adapted from a model previously used to examine meaning negotiation and co-construction of 
knowledge (Gunawardena et al., 1997). In this study, the Interaction Analysis Model was used to measure students’ 
online interactions; the model defines and describes five types of co-construction of knowledge and negotiation of 
meaning. The five types were: 1) the sharing and comparing of information; 2) the discovery of dissonance and 
inconsistency among ideas, concepts, or statements; 3) the negotiation of meaning and co-construction of 
knowledge; 4) the testing and modification of proposed synthesis or co-construction; 5) the agreement statements 
and applications of newly-constructed meaning. 

In addition, the 14-item survey was administered after the online discussion period to measure student 
attitude about cooperative learning methods based on the experiences in the course activities. The survey included 
14 items such as “When I perceived that I as one group member should contribute to my group activity, I 
participated in it actively and positively,” and “I was able to interact with other students more positively and often 
because I had a complementary and interconnected role in my group.” A five -point Likert-type scale was provided 
for the participants to respond to the items as “strongly agree,” “agree,” “undecided,” “disagree,” and “strongly 
disagree.” 

Data Analysis 

One-way analysis of covariance (ANCOVA) with student achievement scores as one dependent variable 
was applied to investigate the differences between Group 1, Group 2, and Group 3. Students posted 2766 online 
messages to the three-week online discussion. One-way analysis of variance (ANOVA) with student interaction (the 
total number of online messages which were posted by each experimental group) as another dependent variable was 
also employed to examine the differences between Group 1, Group 2, and Group 3. 

The researcher investigated the student interactions in each condition by categorizing messages in the 
respective online discussion room. To obtain inter-rater reliability, the researcher and research assistant analyzed 
288 sample data (approximately 10%) from the total of 2766 online messages, which were posted to the three-week 
online discussion. Also, this process used the operations as the specific standard of five categories in interaction 
analysis model. In this study, Cohen’s Kappa was employed to examine the inter-rater reliability for the student 
interaction. In this study, the result of the inter-rater analysis was Kappa = .78, p < .01. It means that this measure 
was statistically significant and indicated a good level of inter-rater reliability. 
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Multivariate analysis of variance (MAN OVA) with the five types of student interactions as the dependent 
variables was employed to examine the differences between Group 1, Group 2, and Group 3. In addition, planned 
orthogonal contrasts were applied to determine which group means between the three groups differed significantly 
from others with respect to the five types of student interaction. In order to analyze characteristics of online 
interactions, the researcher categorized all 2766 online messages which were posted for the three-week online 
discussion into the five types of co-construction of knowledge and negotiation of meaning, which were employed in 
the adaptation of the Interaction Analysis Model (Gunawardena et al., 1997). These data were coded according to 
the following categories: Type 1 — sharing and comparing of information, Type 2 — discovery of dissonance and 
inconsistency among ideas, concepts, or statements, Type 3 — negotiation of meaning and co-construction of 
knowledge, Type 4 — testing and modification of proposed synthesis or co-construction, and Type 5 — agreement 
statements and applications of newly-constructed meaning. 

In addition, a factor analysis was utilized to explore the underlying structure of a set of survey items 
measuring student attitude. This analysis identified two underlying factors: Factor 1 — “Cooperative learning: 
participation and communication resources” and Factor 2 — “Online activities.” Factor loadings and factor scores 
were computed and interpreted with the Principal Component Analysis (PCA) including Varimax rotation method. 
Multivariate analysis of variance (MANOVA) was adopted to examine the differences in the responses of the 
positive interdependence, group processing, and no structure groups with respect to these two factors. In addition, 
planned orthogonal comparisons were applied to determine which group means between the three groups differed 
significantly from others with respect to student achievement, interaction, and attitude in online cooperative learning. 

Results 

Student Achievement 

In Table 2, results indicated that there were significant differences among students in the Positive 
Interdependence, Group Processing, and No Structure groups with respect to their achievement scores. In addition, 
the results of planned orthogonal comparisons showed that participants in the Positive Interdependence groups had 
significantly higher achievement than participants in either the Group Processing groups or the No Structure groups, 
F(2, 140) = 16.15, p <.05. 



Table 2. Tests of the Difference in the Achievement Scores 



Source 


Type I 
SS 


df 


MS 


F 


P 


Partial 
Eta Squared 


Power 


Corrected 

Model 


10279.27 


3 


3,426.42 


209.63 


.00 


.82 


1.00 


Intercept 


966,944.44 


1 


966,944.44 


59,158.81 


.00 


1.00 


1.00 


Fidelity 


9,731.60 


1 


9,731.60 


595.39 


.00 


.81 


1.00 


Group 


547.67 


2 


273.84 


16.75 


.00* 


.19 


1.00 


Error 


2,288.29 


140 


16.34 










Total 


979,512.00 


144 












Corrected 

Total 


12,567.56 


143 













Note. Dependent variable = student achievement. 
* p < .05. 



Student Interaction 

In Table 3, results indicated that there were significant differences among students in the Positive 
Interdependence, Group Processing, and No Structure groups with respect to student interactions. The results of 
planned orthogonal comparisons also indicated that participants in the Positive Interdependence groups and the 
Group Processing groups interacted with each other to a greater extent than those in the No Structure groups, F 
(2,141);= 6.14./; <.05. 
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Table 3. Tests of Group Effect for Student Interaction 





Type III 












Partial Eta 




Source 


SS 


df 


MS 




F 


P 


Squared 


Power 


Corrected 

Model 


1847.17 


2 


923.58 


6.14 


.00 


.08 


.88 


Intercept 


53,130.25 


1 


53,130.25 


353.36 


.00 


.71 


1.00 


group 


1,847.17 


2 


923.58 


6.14 


.00* 


.08 


.88 


Error 


21,200.58 


141 


150.36 










Total 


76,178.00 


144 














Corrected 

Total 


23,047.75 


143 














Note. Dependent variable = the total number of online messages which each student posted. 




* p < .05. 


















Table 4. Tests of Contrasts for the Five Types of Student Interactions 






















Partial 




Source 


Dependent 

Variable 


SS 


df 


MS 


F 


P 


Eta 

Squared 


Power 




Typel 


425.35 


1 


425.35 


22.01 


.00* 


.14 


1.00 




Type2 


160.50 


1 


160.50 


6.47 


.01* 


.04 


.71 


Contrast 1 


Type3 


7.35 


1 


7.35 


0.36 


.55 


.00 


.09 




Type4 


0.17 


1 


0.17 


0.05 


.82 


.00 


.06 




Type5 


32.00 


1 


32.00 


3.55 


.06 


.02 


.46 




Typel 


253.50 


1 


253.50 


13.12 


.00* 


.09 


.95 




Type2 


23.01 


1 


23.01 


0.93 


.34 


.01 


.16 


Contrast 2 


Type3 


140.17 


1 


140.17 


6.78 


.01* 


.05 


.73 




Type4 


7.59 


1 


7.59 


2.42 


.12 


.02 


.34 




Type5 


28.17 


1 


28.17 


3.13 


.08 


.02 


.42 




Typel 


2,724.65 


141 


19.32 












Type2 


3,499.98 


141 


24.82 










Error 


Type3 


2,916.46 


141 


20.68 












Type4 


442.56 


141 


3.14 












Type5 


1,270.27 


141 


9.01 











* p < .05. 



Table 4 shows the results of contrast tests for the five types of student interactions. For Contrast 1, each 
group mean of Group 1 and Group 2 was significantly different from that of Group 3 with respect to Type 1 and 
Type 2 categories (p < .05). However, for Contrast 1, each group mean of Group 1 and Group 2 was not 
significantly different from that of Group 3 with respect to Type 3, Type 4, and Type 5 categories (p > .05). 

For Contrast 2, the group mean of Group 1 was significantly different from that of Group 2 with respect to Type 
1 and Type 3 categories (p < .05). However, for Contrast 2, the group mean of Group 1 was not significantly 
different from that of Group 2 with respect to Type 2, Type 4, and Type 5 categories (p > .05). 

Student Attitude 

The results of MANOVA showed that regarding student attitude towards the experiences of cooperative 
learning: participation and communication resources [T 7 (2, 141) = .17, p > .05], and online activities [F (2, 141) 
= .28, p > .05], there was no significant difference among any of the three groups. 

Discussion 

This study investigated the relative effectiveness of positive interdependence and group processing 
activities on student achievement, interaction and attitude in online cooperative learning. Prior studies on 
cooperative learning have been limited, focusing on either the relative effectiveness of several types and structures 
of positive interdependence on student achievement (Brewer & Klein, 2006; Jensen, Johnson, & Johnson, 2002; 
Johnson & Johnson, 1989) or on the relative effects of some structures of group processing on such achievement 
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(Garibaldi, Johnson, Johnson, & Stanne, 1989; Johnson & Johnson, 2006). Those studies revealed that both positive 
interdependence and group processing had a significant impact on student achievement. The present study built on 
this research by comparing both; examining the relative effectiveness of positive interdependence strategies with 
those of group processing strategies on student achievement in online cooperative learning. The results of this study 
show that using positive interdependence strategies is more effective than using group processing strategies in order 
to improve student achievement. This may result from the fact that positive interdependence has more “diverse” 
instructional methods: i.e. providing students with the strategies of rewards, facilitating students’ learning 
motivation effectively through explaining the specific goals, and implementing the role activities. Group processing 
has relatively simple methods, basically writing periodic processing interim reports. Because of this reason, students 
have more responsibilities to participate in, and contribute to their online activities when receiving positive 
interdependence strategies. 

In addition, positive interdependence strategies are more “structured” and “prescriptive” than group 
processing strategies. During the pre-activities period instructors provide students with goals and explain their role 
activities. During the activities period instructors provide students with appropriate rewards. In other words, positive 
interdependence is focused on improving the final product and incorporates activities during the entire process of 
online cooperative learning, whereas, group processing is focused only on the evaluation of the process of writing 
the interim report, making it difficult to facilitate student motivation effectively. 

The results of this study showed that each condition (positive interdependence and group processing) was 
effective on student interaction. They also showed that there were significant differences in the relative effectiveness 
of positive interdependence, group processing, and no structure on student interaction while completing assignments 
in online cooperative learning environments. The results on student interaction, as measured by the total number of 
online messages posted, revealed that there were significant differences between the means of the positive 
interdependence, group processing, and no structure groups. The results of planned orthogonal comparisons showed 
no indicated differences between the positive interdependence and group processing groups. The results corroborate 
prior studies that examined the usefulness of positive interdependence on student interaction (Brewer & Klein, 2006; 
Jensen et al., 2002; Johnson & Johnson, 1989) and the usefulness of group processing on student interaction 
(Garibaldi et al., 1989). However, further study will be necessary to determine the relative advantages of these two 
strategies on student interaction. 

To examine the differences among the influences on the specific types of interactions, planned orthogonal 
comparisons were employed. While the results of planned orthogonal comparisons indicates a desired effect on 
learning outcome, these types of interactions (“sharing and comparing of information,” and “discovery of 
dissonance and inconsistency among ideas, concepts, or statements”) represent basic involvement with the course 
content and may not be as associated with higher levels of understanding. They also suggest that an instructor’s 
decision to incorporate positive interdependence strategies or group processing strategies can be crucial to student 
performance, especially when the types of student interactions are associated with basic or lower levels of 
understanding. Further research is recommended for to determine the relative desirability of each type of 
interactions in relation to overall instructional goals and the best way of increasing the occurrence of the desired 
interactions. 

A factor analysis was utilized to explore the underlying structure of a set of survey items measuring student 
attitude. The two factors that were identified, “Cooperative learning: participation and communication resources” 
and “Online activities”, show that instructors need to consider 1) how positively students participate in their group 
activities, 2) how they use communication resources, and 3) how they feel about online cooperative learning 
activities while developing and using online cooperative learning strategies. 

Previous research on cooperative learning has also shown that some types and structures of positive 
interdependence (Brewer & Klein, 2006) and some structures of group processing (Garibaldi et al., 1989) increase 
positive student attitude toward cooperative learning. A meta-analysis of 1 1 studies presented some evidence that 
cooperative learning with small online discussion groups had affected undergraduate positive student attitude 
significantly (Springer, Stanne, & Donovan, 1999). The results of this study do not corroborate these prior studies as 
positive interdependence and group processing activities did not show an improvement on student attitude when 
compared to the group with no such activities. However, this difference in findings may be due to the difference in 
the way student attitudes were measured. In addition, this study did not employ as large a subject sample which 
might reduce the statistical power (Howell, 2002). Also, these students generally engage in many online course 
activities that may have a more general effect on their overall attitude and are not changed by one class experience. 
Consequently, further research would be suggested to investigate the additional factors of student attitude, develop 
more appropriate measures of student attitude, and determine the relative usefulness of these two strategies. 
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Implications 

The findings of this study provide positive implications for ways to improve cooperative work in 
undergraduate students who want to know the best ways to participate in online group activities and help them be 
productive and actively contribute to their online group activities. First, these results suggest that instructors should 
be encouraged to consider incorporating cooperative learning formats in their classes and also to incorporate 
strategies into group activities that will enhance student interaction and their resulting achievement. Students in such 
courses should be instructed in the use of these strategies and should be shown the potential benefits of these 
activities early in the course to encourage them to participate fully. Given the choice, positive interdependence 
would be preferred over group processing as it more effectively influences student achievement in online 
cooperative learning environments. Specifically, it is recommended that instructors help students perceive that they 
should actively contribute to, and be more aware of, their online group activities by utilizing positive 
interdependence strategies such as goal-, reward-, role-, and resource-interdependence strategies. In addition, they 
should be prepared to provide students with ongoing feedback about whether they cooperate with each other 
members effectively or not during their online discussion activities by employing group processing strategies by 
having the students submit their group processing reports and post them into their online discussion rooms. 

Second, to take advantage of the benefits offered by Group Processing it is recommended that instructors 
inform groups of the individual progress of each member’s activities periodically and provide students with regular, 
opportune feedback. In other words, group processing might be desired when instructors want to observe students’ 
group activities systematically and obtain better information about how well the students work together (Johnson & 
Johnson, 1994). If students obtain meaningful feedback and perceive that they are evaluated fairly and regularly by 
other group members and by their instructors during their online group activities they may maintain positive 
attitudes about their activities and meaningfully interact with each other to a greater extent. For example, instructors 
are advised to announce that students will be evaluated every week during their online group activities, and post the 
results of their activities in their online discussion rooms In addition, instructors are recommended to respond each 
contribution quickly. They need to post a personal, complementary message to positive contributor (Berge, 1995). 
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The Underlying Reasons of the Signaling Effect in Multimedia 
Learning? Evidence from Eye Movements 

Erol Ozcelik, Ismahan Arslan-Ari, Kursat Cagiltay 

Abstract 

The goal of this study was to examine the effects of signaling on learning outcomes and to reveal the 
underlying reasons of this effect by using eye-movement measures. Forty undergraduate students studied either 
signaled or nonsignaled multimedia materials. The results suggested that the signaled group outperformed the 
nonsignaled group on matching and transfer tests. Eye-movement data showed that signaling directed attention of 
participants to relevant information and reduced unnecessary visual search to find relevant information. 

Theoretical Framework 

Several research studies have shown that learning is enhanced when instructional materials include 
illustrations and narration (Flecther & Tobias, 2005). This phenomenon is called the modality effect. Flowever, 
simply presenting information by using multiple modalities does not ensure superior performance, especially when 
the limitations of the human cognitive system are not taken into account (Sweller, van Merrienboer, & Paas, 1998). 
According to Cognitive Load Theory (CLT), the unnecessary visual search associated with finding the relevant 
information in the diagram will consume some processing resources in the mind (Kalyuga, Chadler, & Sweller, 
1999). Consequently, fewer processing resources will be left for learning (Sweller, et al., 1998). There will be 
impairment in performance as a result of extraneous cognitive load related to the presentation format of instruction. 
One technique that can minimize this problem is signaling, providing cues to students in the most effective and 
efficient way to process the instructional materials (Mautone & Mayer, 2001). 

Although several research studies have examined the effects of signaling on learning, the underlying 
mechanisms of the signaling effect are not clear. Theoretical assumptions of these studies have been mostly based 
on indirect measures such as learning outcomes (Briinken, Plass, & Leutner, 2004). This creates the need to use 
more direct measures to obtain insight about on-line processing of learners. Considering this need, eye-movement 
data such as average fixation duration, total fixation time and fixation count can provide real-time measures of 
cognitive processing during multimedia learning (Flenderson, Brockmole, Castelhano, & Mack, 2007). 

One of the goals of this study was to investigate the effects of signaling on learning from multimedia 
instructional materials. Retention, matching, and transfer tests were administered to measure learning outcomes. The 
other goal of the current study was to find the underlying reasons of the signaling effect by eye-movement data. 

Method 

Participants 

A total of 40 undergraduate students (23 female, 17 male) took part in the study. Participants’ age ranges 
from 10 to 26 years old (.17=2 1 .63, 572=1 .28). Participants were randomly assigned to the signaled (n=20) or to the 
nonsignaled group (n=20). An independent-samples t-test results suggested that there was no significant difference 
between the signaled group and the nonsignaled group in their prior knowledge about the topic, t(38)=1.57,/>=.44. 
Instructional Material 

The 91 second-long Adobe Flash-based narrative instruction was developed by the authors. The multimedia 
package included a labeled illustration of a turbofan jet engine and a narration explaining how turbofan jet engines 
work. The signaled format was identical to the nonsignaled one (see Figure 1) except each corresponding 
terminological label (e.g. nozzle) in the illustration was presented in red color during the narration of the sentence in 
which the item was mentioned. After the narration of the sentence, color of the signaled terminology was returned to 
its original color. The presentation of the computer-based instruction was system-paced. 
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Low-pressure compressor 



High-pressure compressor 



Turbine Compustion champer 



Shat ; 



Turbine vanes Completion dumper 



Figure 1. The nonsignaled format of the instructional material 



Procedure 

Each participant was tested individually in a single session. After giving information about the study, the 
participants were asked to complete the prior knowledge test on turbofan jet engines and a questionnaire on 
demographics. Eye movements of each participant were calibrated automatically by Tobii’s ClearView software with 

5 fixation points. Afterwards, the participants were asked to study the material which was presented only once for a 
fixed total duration of 91 seconds. After studying the instructional material, each participant was administered 
paper-pencil tests consisting of the retention test, the transfer test, and the matching test. Participants were given 
unlimited time to answer the paper-pencil tests. 

Data Collection 

The eye-movement data of the participants were collected by Tobii 1750 EyeTracker while they were 
studying the instructional material. To assess learning, the multiple choice retention test, the open-ended transfer 
test, and the matching test were used. 

Retention test measures how much factual information the learner remember. It includes eight multiple- 
choice questions. Each question consists of five options. Transfer test assesses what extent participants could apply 
the presented instructions to novel problems that were not directly addressed in the material. It consists of five open- 
ended questions, such as “There is not enough air in the turbofan engine. Which components of the engine may not 
be working appropriately?”, 

In the matching test, participants were asked to match the names of the elements (i.e. high-pressure 
compressor, low-pressure compressor, shaft, turbine, fan, combustion chamber, turbine vanes, and nozzle) with the 
parts of the jet engine on a non labeled version of the illustration from study phase. 

Results and Discussion 

A separate independent sample t-test was used to analyze the data. The results suggested that the learners in 
the signaled group had higher transfer and matching scores than the learners in the nonsignaled group, t(38)=2.27, 
p=. 03 and t(38)=2.45, p=.02 respectively. However, both groups performed similarly in the retention test, t(38)=42, 
p=. 68, indicating that the effect of signaling was on deeper processing. Participants engaged in more meaningful 
learning when the relevant visual information with respect to the narration was signaled by color variation (Mautone 

6 Mayer, 2001). 

In this study, the source of signaling effect was further explored by the help of eye-movement data. We 
may argue that enhancement of meaningful learning by signaling had two reasons. First, signaling directed the 
attention of participants to relevant information (Lorch, 1989) which was evident from higher number of fixations, 
t(38)=4.44, /K.001, and longer total fixation time on relevant information including both labels and related parts of 
the illustration, f(38)=3.32,/K.05. Participants spent more time on relevant information when they were guided by 
signals. Second, consistent with previous research on signaling (Kalyuga et ah, 1999) signaling facilitated the 
efficiency and the effectiveness of visual search to find necessary information. Spending less time and having higher 
accuracy to locate necessary information might enable learners to have more time to think about critical information 
to understand the material. Moreover, participants could use more of their processing resources for learning when 
these resources were not consumed by unnecessary visual search (Sweller, et ah, 1998). Learners could engage in 
higher order cognitive processes (e.g. integration) when appropriate signals assisted their cognitive processes 
(Kenneth, 1987). Taken together, these results supported the directing attention and the unnecessary visual search 
hypotheses. Our results that employed measures on relevant information did not rule out the possibility that 
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signaling may affect learners processing of the whole material, rather than the signaled content (Lorch & Lorch, 
1995). This possibility was examined by testing eye-movement measures on the entire material. The differences 
were not significant on fixation count, average fixation duration, and total fixation time on the whole content (all 
ps>.20). This demonstrated that the signaling did not influence the way participants processed the material in 
general. 

Instructional designers should derive benefit from such scientific evidences to create more effective and 
efficient multimedia learning environments. The findings of this study demonstrated that multimedia learning 
materials have great potential in instructional settings. Use of text, illustration and narration in an effective way can 
improve the learning process. Instructional designers should use appropriate signaling cues to help the learner find 
the visual element within the diagram which is related to its corresponding narration (Jamet , Gavo & Quaireau, 
2008). Moreover, users’ attention should be directed to the relevant information during this period of time (Kriz 
&Hegarty, 2007). Especially when the content is complex and the learning environment is system-paced, signaling 
cues are so important ( Harpskamp, Mayer & Suhre 2007). 
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Abstract 

The present study is intended to investigate information literacy of digital natives (those who were born 
after 1980) in the context of learning online and implications of this information literacy on practice and research 
within a Hispanic-serving institution in South Texas, U.S.A. Five latent factors are involved in the study as 
dependent variables. They are behavior pattern (pertaining to technology use), technology ability, learning 
preference, attitude toward technology use, and perception of instructional strategies. In particular, learners’ 
demographics (e.g., age) are factored into this investigation as independent variables. A /-test for independent 
samples suggests that age be the factor to how Hispanic learners perceive (a) the use of technology for course work 
and (b) instructional strategies. 

Introduction 

While digital natives are going to college, they are bringing in values that seem to distinguish them from 
other generation groups (those who were born before 1981). Prensky (2005) states that when compared to the older 
groups (i.e., digital immigrants), digital natives tend to be more causal; they are more inclined to multi-task; they 
have a shorter burst of interest in things. Prensky believes that because this generation group (a.k.a. Net Generation 
or Net-Gen) grows up with all different kinds of digital gadgets (e.g., cell phones, Internet, and computer games) 
around them, the way the Net Gen perceives and behaves is enormously affected by these digital devices and media. 
This argument echoes the work by Tapscott (1997, 2007), who claims these “digital kids” are learning exactly the 
skills that are deemed essential for the 21 st century. 

Apparently, not everybody shares the same optimistic view as Prensky and Tapscott do. Mark Bauerlein 
(2008), author of the book, titled The Dumbest Generation: How the Digital Age Stupefies Young American and 
Jeopardizes Our Future, contends that the Net Generation in the United States may be the dumbest generation, 
compared to their parents and other older groups. But, the truth is, according to the U.S. Department of Education’s 
National Assessment of Educational Progress (NAEP) (as cited in Howe, 2008), the lowest scores on mathematics 
and reading tests ever recorded by NAEP were reported by those who were bom between 1961 and 1965. These 
children are now in their late 40s. And, most of them are parents of the Net Generation. 

Whether they are the smartest generation or not, these digital natives are now sitting in the classroom. As 
professors, we ought to attend to any differences in values between this new generation and others before, in 
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particular, information literacy in the digital era. Information literacy (for the 21 st century workforce) generally 
includes, as Speak Up 2008 national research report (by Project Tomorrow, an education non-profit organization 
based in Irvine, California, U.S.) suggests, communications, collaboration, creation and contribution. Often time the 
majority of people (including university professors) would conveniently assume that the digital natives are equipped 
with great technology skills, which enable these young learners to better search, organize, evaluate, and share any 
given information. Is this true? If so, does this assumption also hold true for Hispanic students? This is what this 
investigation is aiming at. 

Background 

Prensky (2001) coins two terms, digital natives and digital immigrants. The former is a take-off on the 
phrase, native speakers. According to Prensky, digital natives refer to those individuals who interact with the 
computer and video games to an extent equivalent to speaking the first language whereas digital immigrants are 
those who interact with the computer and games in a fashion similar to learning to speak a foreign language. He 
argues that because digital natives grow up with the frequent interaction with the technology their cognitive thinking 
seems to be programmed in a differing fashion than their classmates who were born earlier. Prensky (2005) claims 
that these younger learners are now able to achieve learning through a cellure phone, which is one significant 
activity or behavior in their digital life. As more and more digital natives attend college, a debate whether or not 
university professors ought to restructure their instruction in a hope to accommodate these digital natives has 
received further attention (Carlson, 2005). 

Following upon the previous thought of digital natives entering the college, literature shows the Net 
Generation (those at the age between 21 and 25) or the Net Gen tends to demonstrate stronger abilities of using 
technological devices in their daily activities than their peers (Gracia & Qin, 2007). Garcia and Qin’s study in 2007 
indicates that technology ability is a factor that clearly distinguishes the Net Gen learners from their fellow students. 
As far as student’s perception of instructional strategies, there is not any significant difference between the two 
groups. For instance, the surveyed 280 students report that university-level courses are effective if they are taught 
via instructor’s lectures and discussions with the faculty. 

What about this age group of students’ learning preferences for campus technology and attitudes toward 
technology use? With 4,374 college students (those at the age of 25 or younger) from 13 institutions of higher 
education within five states of the United States of America surveyed, a study by Kvavik (2005) reports that 41% of 
these students seem to have a preference for classes using moderate information technology (IT), 2% favor fully 
Web-based classes, and 3% prefer a class without any IT. The large-scale study also suggests that this preference for 
IT is positively correlated with students’ prior experience with the classroom technology and professors’ technology 
skills. So far as ethnicity is concerned, there is little generational difference research that indicates the findings 
aforementioned hold true across races or ethnic groups (Reeves, 2008). For this matter, the present investigation 
purposively targets Hispanic learners, specifically Mexican American students. Four research (alternative) 
hypotheses are formed for the present study: 

Hypothesis 1 : There is a significant difference in means between age groups in terms of technology ability. 

(H 0 : There is not any significant means difference between age groups in technology ability.) 

Hypothesis 2: There is a significant difference in means between age groups in terms of learning preference. 
(H 0 : There is not any significant means difference between age groups in learning 
preference.) 

Hypothesis 3 : There is a significant difference in means between age groups in terms of learner attitude 
toward technology use. (H 0 : There is not any significant means difference between age 
groups in learner attitude toward technology use.) 

Hypothesis 4: There is a significant difference in means between age groups in terms of instructional 
strategies. (H 0 : There is not any significant means difference between age groups in 
instructional strategies.) 

The results of this study are expected to inform online course developers and cyber instructors of potential 
instructional strategies that may be taken into account in the design and development of any course with an online 
component integrated. Knowing that the disagreement on the digital natives previously mentioned in the 
introduction, this inquiry may be able to shed some light on how these Net Gen students perceive their online 
learning, compared to the digital immigrants, and how this perception may be worth further effort by designers and 
instructors to better prepare their instructional materials. It is also anticipated that their students are able to further 
reflect on their perceptions, attitudes, and behaviors pertaining to information literacy and better enjoy their learning 
experience in an online environment in light of this study. 
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Method 

Design 

This quantitative inquiry centers on information literacy in the Hispanics’ perceptions of online learning 
and their implications on the development of eleaming (i.e., online learning). Targeting a Hispanic-serving 
institution in the U.S., learners’ age difference is factored in to the study, along with five other factors: behavior 
pattern (pertaining to technology use), technology ability, learning preference, attitude toward technology use, and 
perception of instructional strategies. 

Setting 

The setting of this survey research is located in the online learning environment within a state university in 
South Texas, U.S.A. According to Santiago (2006), the university is a Hispanic-serving institution (HIS) with 
approximately 94 percent of total undergraduate Hispanic full-time equivalent student enrollment. 

Participants 

Target population is the entire student body (both graduate and undergraduate) that is enrolled in any class 
that has an online Blackboard (a course management system) component in it. Accessible population is those whom 
investigators are given instructor’s permission to survey. The surveyed students must be at least 18 years old and 
participate in this study on a voluntary and anonymous basis. As suggested by Hartman, Moskal, Dziuban (2005), 
participants are divided into three age groups: (a) those bom before 1965 (aka Baby Boomers), (b) those born in 
1965 through 1980 (aka Generation X), and (c) those born after 1980 (aka Net Gen). 

This study was first launched in the summer of 2008. As of March 2009, 147 students have successfully 
responded to the questionnaire. Of all, 110 are Hispanic learners, with 50 identified as Non Net Gen and 60 as Net 
Gen. Please note that due to the sparse distribution of the baby boomers group, the three age groups are collapsed 
into the dichotomy. Non Net Gen and Net Gen. See the data analysis section below for more information. 

Data Collection Procedures and Instruments 

Data are collected using a password-protected computer server through an online questionnaire. The 
questionnaire is composed of six instmments: 

1 . Demographics Instillment (9 questions) on a nominal scale 

2. Behavior Pattern Instalment (23 questions) on a nominal scale 

3. Technology Ability Instalment (25 questions) on an interval scale 

4. Learning Preference Instalment (27 questions), adapted from the work by Loo (2004) on an interval scale 

5. Attitude Instalment (20 questions), adapted from the work by Ajzen and Fishbein (1980) on an interval 

scale 

6. Instaictional Strategy Instalment (8 questions), adopted from the work by Tapscott (1997) on an interval 

scale 

The four instalments on the interval scale are on a five-point Likert scale with “1” as strongly disagree, “5” as 
strongly agree, and “3” as neutral. The questionnaire is administered at one occasion earlier in a semester, which 
begins in the third week and end in the fifth week of a regular semester (fall and spring). In any summer semester, 
the administration begins in the second week and end in the fourth week. With instructor’s prior approval, 
investigators email the consent form through Blackboard email system. Data sets will be purged from all the 
computer hard drives in five years after the research begins (by the end of spring 2013). 

Data Analysis 

Data sets are exported from the password-protected computer server to Microsoft Office Excel before 
entered to the SPSS program. Due to the number of the participants, the baby boomers group and the Generation X 
group are combined. The merged group is then named, Non Net Gen. T-test for independent samples is used in 
addition to reliability testing. 
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Results 

The table below shows descriptive statistics by variable and group. 



Summary Statistics 


Variable 


Group 


N 


Mean 


Std. Deviation 


Std. Error Mean 


Technology 

Ability 


Non Net Gen 


50 


1.0702E2 


16.51652 


2.33579 


Net Gen 


60 


1.0240E2 


16.56850 


2.13898 


Learning 

Preference 


Non Net Gen 


50 


97.98 


13.797 


1.951 


Net Gen 


60 


94.38 


15.245 


1.968 


Learner Attitude 


Non Net Gen 


50 


92.34 


8.198 


1.159 


Net Gen 


60 


85.17 


14.039 


1.812 


Instructional 

Strategy 


Non Net Gen 


50 


25.56 


4.554 


.644 


Net Gen 


60 


22.95 


4.986 


.644 



Hypothesis 1 

The research hypothesis here is, “There is a significant difference in means between age groups in terms of 
technology ability.” An independent samples t-test indicates that there is not any scientifically significant difference, 
t(108) = 1.46 ,p = .15, suggesting the alternative hypothesis is rejected in favor of the null hypothesis. 

Hypothesis 2 

The research hypothesis here is, “There is a significant difference in means between age groups in terms of 
learning preference.” An independent samples t-test indicates that there is not any scientifically significant 
difference, t{ 108) = 1 .29, p = .2, suggesting the alternative hypothesis is rejected in favor of the null hypothesis. 
Hypothesis 3 

The research hypothesis here is, “There is a significant difference in means between age groups in terms of 
learner attitude toward technology use.” An independent samples t-test indicates that there is a scientifically 
significant difference, 1(108) = 3.19 ,p = .002, suggesting the alternative hypothesis is accepted. 

Hypothesis 4 

The research hypothesis here is, “There is a significant difference in means between age groups in terms of 
instructional strategies.” An independent samples 1-test indicates that there is a scientifically significant difference, 
1(108) = 2.84 , p = .005, suggesting the alternative hypothesis is accepted. 

Conclusions 

Intended to better inform concerned instructional designers and online instructors, the current phase of the 
investigation on generational differences in information literacy is able to collect more participants from both 
graduate and undergraduate students than the earlier one. Of all the 147 participating students, 1 10 Hispanic learners 
are divided into two independent groups, Net Gen and Non Net Gen and further analyzed. The four instruments on 
the interval scale adapted from the literature appear reliable as the reliability testing suggests. Respectively, Alpha 
values are .94 for Technology Ability Instrument, .89 for Learning Preference Instrument, .95 for Attitude 
Instrument, and .76 for Instructional Strategy Instrument. There is not any significant difference between Non Net 
Gen and Net Gen in their perception of self technology abilities (e.g., use of email and the WWW) or their 
preference to instructional activities (e.g., group discussions and individual reflection writing). Nevertheless, there is 
a significant difference between Non Net Gen and Net Gen in their attitude toward technology use (e.g.. Blackboard 
and electronic devices used for coursework) and their preference of instructional strategies (e.g., teacher-centered vs. 
learner-centered). The Net Gen group seems to favor an instruction that is more structured, and more instructor- 
centered. This finding is not supported by the literature (e.g., Prensky, 2005). This may have to do with the fact that 
the majority of the participants are freshman and graduate students. Almost all of these surveyed freshmen are Net 
Gen whereas most of the surveyed graduate students are Non Net Gen. Besides, these graduate students are enrolled 
in some fully-based class, and these freshmen are not. Perhaps these discrepancies result into the finding. Further 
research should probe into the generational differences on the manifest variable level. 
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Layout, Structure, and Semantic Embedding. On the Problem of Finding 
Accessible Features for the Evaluation of the Quality of Learning Texts. 
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Abstract 

The fact that the quality of a text influences learning outcome is no surprise. Better text will 
certainly evoke better learning, and it is not hard to find general advice on writing a good text. On 
the empirical basis of semantic, syntactic, and layout text features we conducted a study to find 
out more about the text features which we believe to cause perfonnance gain and loss. Within the 
data analysis we show that the prediction of text quality is far more complex than it is usually 
assumed to be. Common qualitative categories and quantitative features both could not explain 
effects which text has on learning outcome. Additionally, we found out that the subjective 
perception of the learner on several important text quality measures has no effect on learning 
perfonnance. This is an important lesson to learn - especially for the evaluation of texts which are 
meant for learning. 

Keywords: Text Quality: Reading Comprehension; Semantics; Syntax; Layout; Learning and Instruction 

Text Quality influences Learning Outcome 

There is plenty of evidence that text quality can influence learning (Goodman, 1967; Jones, 2005; 
Townsend et al., 1979; McNamara et al., 1996; Kintsch & van Dijk, 1978; Just & Carpenter 1976; Stevens, 
1980; Deuze, 2005; Skinner et al., 2001). In a previous study (Pirnay-Dummer & Ifenthaler, under review) we 
found out that some texts, though intended specifically for learning, may even systematically decrease learning 
performance. We assumed that the influence on learning outcome had to come directly from text features. 
Choosing the features outlined below, we designed a new study to shed light on the text features we believe to 
influence performance. 

Semantic Features and the Semantic Embedding Indices 

The question of what a text is about and how information is carefully connected is of key interest when 
we think about learning from texts. In linguistics, the field of semantics provides strong implications on the 
generation of meaning from natural and written language. According to Leech (1974), “semantics (as the study 
of meaning) is central to the study of communication” and “also at the centre of the study of the human mind - 
thought processes, cognition, conceptualization” (Leech, 1974, p. ix). This idea starts with identifying the 
smallest possible increment, the letter, and then broadening the perspective to include phrases, sentences, texts, 
and even entire discourses involving world knowledge or more information than the actual text itself may 
provide (Bach, 1989; Schwarz & Chur, 2007). Investigation of semantic features has enjoyed growing interest 
among linguists since the 1970s, especially in the area of model-theoretic semantics (Schnotz, 1987). 

In this study, we will focus exclusively on text semantics with a model-theoretic background. The 
semantic features believed to facilitate the understanding of a text can be separated into three major areas: 
Cognitive structure, cognitive dissonance, and semantic redundancy. Two of these areas can be further 
subdivided as shown below. 

Cognitive structure 

The cognitive structure describes the systematic use of the reader’s mental models, their restructuring, 
or additions to them. 

Summary. A summary will help the reader to either see what the text will be about or to reconstruct its main 
arguments (Groeben, 1982). Summaries can be presented in the form of introductions, abstracts, advance 
organizers, or a classical post-text wrap-up. Ballstaedt (1997) argues that every text consisting of more than 
2,500 words should contain a summary of some sort to facilitate understanding. 

Structure. A well-structured text will provide orientation for the model a text provides and thus 
facilitate the restructuring of the reader’s own models. This structure can take on a wide range of forms: 
inductive or deductive (Seel, 2003b), chronological, thematic, or hierarchical (Anderson, 2001) to name a few. 

Examples. With the help of examples, the reader may be able to use his world knowledge (Seel, 1991) 
to understand unknown concepts in the text more easily. Examples may incorporate a simple analogy to known 
facts, a recapitulation of previously stated similar ideas, or a physical phenomenon in the real world. 

Foreign words. Words unknown or at least obscure to the reader will not only slow down the reading 
process (Ballstaedt, 1997), they will also impair the creation of coherence (Groeben & Christmann, 1999) and 
inference structures (Seel, 2003a), as they do not correspond to objects in previous experience. In this study a 
foreign word is operationalized as a rarely used expression or abbreviation unless it is a proper noun 
representing a unique entity. 
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Explanation of foreign words. Heavy and frequent use of foreign words may be mitigated by an 
explanation of these words to facilitate reader inference. Of course, as with examples, these explanations have to 
incorporate prior knowledge to be effectively added to the reader’s mental model (Seel, 1995). 

Cognitive dissonance 

Cognitive dissonance describes the destruction and rebuilding of the reader’s mental models. Not only 
will the reader more easily remember the elements that caused the dissonance, he will also be more motivated to 
learn more from the text (Seel, 2003b). 

Change of perspective. Adding different perspectives to a topic will not only make the text less biased, 
but also more rich and rewarding for the reader (Groeben & Christmann, 1999). By questioning certain facts 
stated in the text, the reader is forced to challenge the models already provided and thus to either accommodate 
or assimilate the new information. Change of perspective may also add to semantic redundancy (see below). 
Novelty. Offering new areas inside the text will enable the reader to perceive more elements of the topic, to 
cross-reference between them, and ultimately to construct a much more advanced model (Seel, 2003b). 

Questions and consolidation. A text which raises questions will enable the reader to take a more critical 
look at the topic. In contrast to a mere change of perspective, questions force the reader to seek for the answer 
inside his or her mental models from the outset (Groeben & Christmann, 1999). If these questions are later 
answered by the text, the reader will be able to confirm the answers previously provided and consolidate his or 
her model of the areas covered. 

Semantic redundancy 

Semantic redundancy describes repetition of a previous statement or concept in a different way. 
Basically, this means explaining the same ideas with different words several times. According to Groeben & 
Christmann (1999), this can help reinforce the main ideas of the text. As stated by Groeben & Christmann 
(1999), semantic redundancy will only have an effect if the repetitions are also syntactically well structured. 

Syntactic Features and their Main Indices 

Syntax is one of the most important features for rating a text and making it useable from the 
perspective of comprehension (Urban, 1994). Text characterization may be based on a number of syntactic 
features. For example, there are distinctions between word classes and classifications of sentence parts for 
defining their position in a sentence (Merten, 1995). However, analyses on the level of individual words and 
sentences are difficult to realize, especially for longer texts such as instructional texts, in which one has to invest 
a lot of time and cost to identify syntactic features (Merten, 1995). With the average number of words per 
sentence it is possible to measure sentence length. The longer a sentence is, the more complex its structure will 
be, making it more difficult to understand content coherence (Ballstaedt, 1997). Syntactically complex sentences 
measurably impede fluent reading and understanding; they require more processing efforts and may lead to the 
establishment of incorrect content references and overload the working memory (Ballstaedt, 1997). Word 
length and sentence length have demonstrable effects on the understanding of texts (Hartley, 1985; Bamberger 
& Vanecek, 1984). In nested sentences, for instance, it is very difficult for the reader to establish coherence. The 
short term memory is overloaded with overly long and frequent insertions. Hence, sentence length can serve as a 
rough indicator for sentences which are difficult to understand (Ballstaedt, 1997). To help identify texts with too 
many syntactically complex structures, there are readability and comprehensibility formulae which measure 
word length and sentence length as indicators for the readability of a text (Groeben, 1989; Ballstaedt, 1997). 

The Amstad Index 

One of these formulae, the Reading Ease Index developed by Flesch in the year 1948 (e.g. Groeben, 
1989), is still in use today. It was developed to rate English texts and is calculated via the number of syllables 
per 100 words and sentence length. An index of 0 means factually illegible or very difficult to read, and a factor 
of 100 means best readability (Bamberger & Vanecek, 1984). However, a transfer to German does not work 
very well (Groeben, 1982). Amstad adapted the Flesch Reading Ease Index to the German language in the year 
1978 (Bamberger & Vanecek, 1984). The text which had a negative impact on learning in our previous study 
also had a low score on the Flesch index. The Amstad Comprehensibility Formula is calculated as follows 
(Ballstaedt, 1997): A sample from a text is taken, containing a minimum of 100 words. The word length (wl) is 
calculated via the average number of syllables per word. To this end, an online tool which automatically counts 
syllables is used (Bachmann, 2007). Before using the tool for our study, we tested it on several sections of text 
and counted the syllables with and without using the tool. The error rate of the tool was below 5%. In a third 
step, the average sentence length (si) is calculated as the average number of words per sentence. We counted 
this using a tool in Microsoft® Word. Finally, the calculated values are computed with the following equation: 

CF = 180 - (si + wl x 58.5). 

As in the Flesch Index, the accessible values range from 0 to 100. The higher the value, the easier the 
text (Ballstaedt, 1997). 
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Parentheses 

Further investigation of the syntactical features of the texts used in our previous study unveiled another 
important feature that influenced learning negatively. The text with a negative impact on learning had a 
particularly high number of parentheses in comparison with the other two texts used in the study. Parentheses 
interrupt the main sentence and thus may also interrupt reading fluency (Schneider & Raue 2003; Ballstaedt, 
1997); they have a distracting effect while reading a text. Hence, another syntactical feature we varied in our 
study was the occurrence of parentheses. 

Features of Layout with Resulting Indices 

When we think about text layout, we may think about books and other print materials which have to 
fulfill aesthetic demands. According to Bamberger and Vanecek (1984), reading can be described as figuring 
out sense from written signs (such as words), but understanding is also influenced by other features, among 
them text layout (Bamberger & Vanecek, 1984). As pointed out by Sauer (1995, 1997), a common mistake in 
editing a text and designing its layout is trying to make it look “aesthetic” without regard to the integrated 
concept of the text. It is important to take the aspect of layout seriously, as it is not just a matter of creativity or 
the pleasure of graphic designers. 

From a background of didactics it can be argued that layout is not only a way to support the marketing 
of texts. The layout’s power to influence attention, motivation, and remembrance also may be expected to have 
an impact on higher cognitive learning functions (Ballstaedt, 1997). In our study the focus was on aspects of 
macro-typography rather than micro-typography as features like font, font size, line spacing, etc. were identical 
among the texts in the previous study. In comparing the layout of the three texts, we found three features which 
were different: the text length in words, the number of headings per page, and the number of paragraphs per 
page. 

Headings per page 

Hartley (1985) points out that headings support search, recall, and retrieval and thus play an important 
role in reading and furthermore in understanding and learning text contents. In addition, Groeben (1982) 
emphasizes the role of headings as a way to summarize information in the text and stabilize concepts. 

According to Ballstaedt (1997), a heading is a short and introductory fonnulation set before a paragraph. Its 
purpose is to ease the reader’s access to the text and provide a concise key word or key sentence so that the 
reader receives an idea of what the text, and especially the following paragraph, is about. One may differentiate 
between formal, perspective, and thematic headings. In our study the focus was on thematic headings (like in the 
previous study). A thematic heading provides a central word or core statement concerning the following 
paragraph and helps the reader create a macro-structure for his or her long-tenn memory (Ballstaedt, 1997). 

Number of paragraphs per page 

Another important feature of text layout is the structuring in paragraphs (Bamberger & Vanecek, 1984; 
Ballstaedt, 1997). On the one hand, a text with too small a number of paragraphs may appear monotonous, 
tedious, and unhelpful for comprehension and learning. Overloaded pages can also cause discouragement and 
thus inhibit the motivation to read. On the other hand, too many paragraphs interrupt the reader’s train of 
thought and can disturb his or her acquisition of the text’s sense (Bamberger & Vanecek, 1984). It thus seems 
logical that it is useful to find a middle road between too many and too little paragraphs. 

Text length 

Langer et al. (1993) defined length and verbosity as another striking attribute of text layout. Texts that 
are too short or too compact can inhibit understanding as well as those that are too long and long-winded. 

Langer et al. (1993) recommend taking the middle road between briefness and verbosity. 

Research Question 

The study’s main purpose was to investigate the influence of the theoretical assumptions from 
semantics, syntax, and layout on learning outcome: 

How can easily accessible semantic, syntactic, and layout features predict the learning outcome of a learning 
text? And: How can these measures be applied to evaluate texts? 

Hypotheses 

Accordingly, our hypotheses cover the influence of semantics, syntax, and layout on learning outcome 
as well as their interactions. 

Hu: Texts with better semantic embedding indices lead to better learning perfonnance. 

Hj 0 : Texts with better semantic embedding indices do not lead to better learning perfonnance. 

H 2 .i: Texts with better syntactic indices lead to better learning perfonnance. 

H 2 0 : Texts with better syntactic indices do not lead to better learning perfonnance. 

H 3 j: Texts with better layout indices lead to better learning perfonnance. 

H 3 0 : Texts with better layout indices do not lead to better learning perfonnance. 
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Since semantics and syntax are not independent of one another, we expect minor interaction effects 
between the two indices. Analogously, we do not expect interactions with layout features. 

Methods 

The study was conducted with instructional design students in their second semester. They had no significant 
prior knowledge in any of the subject domains. 

The content was chosen from the diverse subject domains of medicine, physics, history, law, sports, 
biology, technology, and society. The texts were taken from the online resource Wikipedia and altered 
afterwards to selectively resemble the feature sets under investigation. Like in the previous study, the subjects 
were given a declarative knowledge test prior to and after reading. Additionally, the subjects filled out the 
general text understanding questionnaire which was also used in the original study. The test was constructed on 
the theoretical basis provided by Groeben (1982) and Langer et al. (1974). All items within this test are 
measured on five-point Likert scales. The four scales of the test are: 

Simplicity [12 items] (e.g. ease of reading), order [12 items] (e.g. structure and design), length [13 
items] (e.g. appropriateness of length), and motivational aspects [8 items] (e.g. mood of the text, writing style 
acts as stimulant). 

Semantic features 

Each text was qualitatively analyzed via a systematic content analysis for the semantic features listed 
above. Features were rated on a five-point scale: Texts that incorporated strong presence of a feature were 
awarded five points for that feature, texts that hardly ever incorporated it were awarded only one point. For the 
feature “foreign words ” the scale had to be reversed, i.e. five points for none to very few foreign words, one 
point for too many. Additionally, there was an option for zero points if any of the features was not present in a 
text at all. Some texts were manipulated by deleting or adding foreign words, explanations of foreign words, 
examples, and summaries to further improve or impair their overall semantic embedding. A main feature’s point 
value was calculated via the median of its sub-features; the total point value of a text was calculated via the 
average of the main features’ point values. Texts with a point value of <2.3 were considered semantically 
lacking, those with a point value of >3.7 were considered semantically well embedded. These limits were set 
taking into account the semantic embedding and test results of the texts used in the previous study. 

Syntactic features 

In the previous study, the text which had a negative impact on learning had a Flesch Index of 3 1 .6, 
whereas the second best text had an index of 34.8 and the best one had an index of 45.3. Consequentially, we 
assumed that texts with an Amstad Index of 30 or less could be characterized as syntactically poor, whereas the 
syntactically well-designed texts would have an Amstad Index ranging from 48 to 60, assuming from the 
previous study that an Amstad Index of 45 or higher characterizes syntactically well-designed texts. To vary the 
Amstad indices of the texts used in the study we elongated the text’s sentences by combining short sentences. 
Long sentences are one variable which influences the Amstad Index of a text, while word length, which is the 
other variable, cannot be varied without major interference with the other features. In checking the number of 
parentheses per page, we found that the text with negative results in the previous study had the most parentheses 
per page of the three texts used in the previous study with an average of 7.5 per page, while the best text had an 
average of only 3.25 per page and the second best text 6.5 (including dates). 

Hence, in our study we varied the number of parentheses per page. Few parentheses per page are assumed to 
enhance reading ease while many parentheses reduces reading ease. Few parentheses were operationalized with 
a number of parentheses per page of < 3.5. Many parentheses per page were operationalized as < 8.4 per page. 
These values were based on the previous study. 

Layout features 

By analyzing the three texts used in the previous study, we figured out that the text with the worst 
learning outcomes had an average number of 1.6 headings per page and the text with the best results 2.8 per 
page. Thus, we modified the texts in such a way that a value for good layout was 2.5 to 3 headings per page and 
for bad layout 1.5 to 1.7. 

With reference to the three texts from the previous study, the text yielding the worst outcomes had an 
average number of 7 paragraphs per page whereas the text with the best outcomes had an average of 5.4. Using 
these facts we determined that an amount of about 7 paragraphs per page is an index for bad layout and 5 per 
page an index for good layout. The text from the previous study with the best learning outcomes had about 1800 
words and the one with the worst results a length of about 3000 words. We thus chose texts which ranged from 
1800 words as an indicator for good layout and about 3000 words as an index for bad layout. 

For every text property set (semantics, syntax, and layout) there were features which were derived from 
the original study. Each property set was treated as a joint factor of all its subsuming features. All features of a 
set were explicitly either favorable or unfavorable, leading to a 2x2x2 factorial design: 
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Table 1 . One Text for each Factorial Condition 

Semantics + Semantics - 



Layout + Syntax + 1 Text 1 Text 

Syntax - 1 Text 1 Text 

Layout - Syntax + 1 Text 1 Text 

Syntax - 1 Text 1 Text 



The eight fields had one text each which fulfilled the property set, e.g. there was a text with high semantic 
embedding, low syntactical quality, and high layout quality. 



Table 2. Feature Variation between the Texts 



Text 


Layout 


Syntax 


Semantics 


Colorectal Cancer 


- 


- 


- 


Stellar Evolution 


+ 


- 


+ 


Battle of Badr 


+ 


+ 


+ 


Guardianship 


- 


- 


+ 


Lacrosse 


- 


+ 


+ 


Polymerase Chain Reaction 


- 


+ 


- 


Stereoscopy 


+ 


+ 


- 


Urbanization 


+ 


- 


- 



Each text featured three questionnaires, one for pretesting subject knowledge with six questions on the 
text, one for testing reading motivation, and one for post-testing subject knowledge with randomized pretest 
questions. The knowledge tests were rated in a simple point system, correct answers being awarded with one 
point, wrong answers with none. The mean pre-post-test difference was calculated for each test. Every subject 
went through three learning cycles with a different text each time. All texts, features, and positions were 
systematically balanced to exclude position effects in all factors. 

Results 

Like in the previous study, the texts themselves had the most important influence on the learning outcome. 



Differences between the Performance Gains 




Figure 1. Differences between the performance gains by the texts 

The differences between the pre- and post-test results of the texts were statistically significant with a 
strong effect (Text, F=2.2622, df=7, df res =112, p < 0.05, r) 2 =0.141). 

Furthermore, none of the three categories (semantics, syntax, and layout) yielded a significant pre -post-test 
difference by themselves. 
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Figure 2. No significant differences in performance gains 



For our core hypotheses in this study we have to keep all three null hypotheses: 

H] 0 : Texts with better semantic embedding indices do not lead to a better learning outcome. 

H 20 : Texts with better syntactic indices do not lead to a better learning outcome. 

H 3 0 : Texts with better layout indices do not lead to a better learning outcome. 

Only the interaction between layout and syntax yielded a significant difference (Layout:Syntax, F=5.9491, df=l, 
df res =l 12, p < 0.05, r| 2 =0.053). 




This interaction has a medium effect and works in such a way that strong syntax combined with weak 
layout results in better learning performance (pre -post-test difference means) as does weak syntax combined 
with strong layout. 

So far, it seems that text quality is determined by idiosyncratic text properties rather than by systematic 
semantic, syntactic, and layout features. 

Post-Hoc Analysis 

Interestingly, the features do have effects on how the text is perceived by the learners - as rated on the 
general text understanding questionnaire. As mentioned above, all subjective ratings are rated on multiple-item 
five-point Likert scale where 5 indicates the best quality and 1 the worst. 

Perceived order 

First of all, the semantic features influenced the perceived order (structure and design) of the text. 
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Figure 4. Semantic features influence the perceived order 

The texts with strong semantic features were perceived as more orderly than those with weak 
semantics. The influences of all three text quality dimensions are distributed as follows. 
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Table 3. Semantics, Syntax, and Layout Influence the Perceived Order (Means) 

Weak Syntax Good Syntax 

Weak Semantics Good Semantics Weak Semantics Good Semantics 

Weak Layout 3.22 3.90 3.05 3.71 

Strong Layout 3.92 3.66 3.43 3.81 



The differences from semantics are statistically significant within a multi-factor ANOVA (Semantics, 
F=8.2616, df=l, df res =l 12, p<0.05, r| 2 =0.074). Layout and semantics had also a small interaction effect 
(Layout:Semantics, F=5.5805, df=l, df res =l 12, p < 0.05, r| 2 =0.05). 

Perceived simplicity 

The perceived simplicity measures how accessible the learners find the text. We also tested all three 
dimensions for this scale. 

Table 4. Semantics, Syntax and Layout Influence the Perceived Simplicity 
Weak Syntax Good Syntax 

Weak Semantics Good Semantics Weak Semantics Good Semantics 

Weak Layout 2.38 3.51 2.87 3.74 

Strong Layout 3.68 3.02 2.96 3.69 

There were high effects on the perceived simplicity from semantics (Semantics, F(l,l 1 2)=1 6. 1303, 
p<0.05, T) 2 =0.144) and the interaction between layout and semantics (Layout:Semantics, F(l,l 12)=12.799, 
p<0.05,r| 2 =0.114). 
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Figure 5. Semantic features influence the perceived simplicity (means). 

Additionally there were interaction effects between semantics and syntax (Semantics:Syntax, 

F(l,l 12)=4.7358, p<0.05, r| 2 =0.042) and between all three feature dimensions on the perception of text 
simplicity (Layout:Semantics:Syntax, F( 1,1 12)=9.9174, p<0.05, r| 2 =0.089). 

Perceived length 

All three text quality dimensions were also tested for perceived length. 

Table 5. Semantics, Syntax, and Layout Influence the Perceived Length (Means) 

Weak Syntax Good Syntax 

Weak Semantics Good Semantics Weak Semantics Good Semantics 
Weak Layout 2.73 05 Z74 09 

Strong Layout 3.41 2.87 3.08 3.34 

For the perception of whether a text had a proper length, the interaction between layout and semantics 
was statistically significant (Layout:Semantics, F( 1 ,1 12)=8.5077, p<0.05, T) 2 =0.076) as was the interaction 
between all three factors (Layout:Semantics:Syntax, F(l,l 12)=6.5315, p<0.05, T) 2 =0.058). 

Perceived motivation/stimulation 

All three dimensions were also tested for their influence on the scale for perceived motivation and stimulation. 

Table 6. Semantics, Syntax, and Layout Influence the Perceived Motivation (Means) 

Weak Syntax Good Syntax 

Weak Semantics Good Semantics Weak Semantics Good Semantics 
Weak Layout 2.45 2A8 162 199 

Strong Layout 3.08 2.93 3.08 2,93 



Whether a text was rated as being motivating and stimulating depended only on the interactions 
between layout and syntax (Layout: Syntax, F=5.4955, df=l, df res =l 12, p<0.05, r| 2 =0.049) and between 
semantics and syntax (Semantics:Syntax, F( 1,1 12)=5. 4364, p<0.05, T| 2 =0.049). 

As a simplification, we can state that good syntax goes better with weak semantics and the other way around. 
Again, good syntax goes better with weak layout to induce good motivation. 

Learners subjective text rating and learning performance 
To see whether the learners’ text quality perception corresponds to their real learning performance, we 
tested a multiple regression model. Flowever there is no correlation between any of the learners’ ratings and the 
real learning outcome (R 2 cor r = -0.03). The learner’s performance cannot be predicted by how positively he or she 
rates the text (F(4, 1 15)=0. 1271 , p>0.05). Whether a learner finds a text to be in good order, stimulating, of 
reasonable length, or simple enough does not have anything to do with how he or she gained performance from 
the text. 
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Text rating per text 

The rating of text quality depends significantly on the text itself, and high effects can be shown on all four 
scales: 

• Text x Order: F(7,112)=3.1424, p < 0.05, if=0.196 

• Text x Simplicity: F(7,l 12)=7.2018, p < 0.05, r| 2 =0.450 

• Text x Length: F(7,l 12)=3. 2648, p < 0.05, r| 2 =0. 204 

• Text x Motivation/Stimulation: F(7,112)=2.6504, p < 0.05, r| 2 =0.166 

This can be seen as indicator for the discriminatory power of the ratings. 

Discussion and Conclusion 

This study clearly shows that the prediction of text quality is far more complex than it is usually 
assumed to be. 

If features and indices other than pure content influence the performance which may be expected from a 
learning text, then they will have to be identified in further studies. If content were the only criterion, it would 
not be possible to design text quality by any means. This is, however, still very unlikely. But yet, the theory- 
based common features discussed here (semantic, syntactic, and layout) did not help at all to predict the learning 
outcome, although they are all commonly used as criteria to evaluate texts for learning. The somewhat 
counterintuitive results of this study also show that simple evaluation methods - especially learners’ ratings - 
can not be used to make stable predictions of learning outcome, nor can the typically applied semantic, 
syntactic, and layout features: The qualitative categories and the quantitatively oriented features both failed to 
influence the transfer quality of the texts. Our post-hoc analysis indicates that semantic quality had an especially 
strong influence on how good a text is perceived as being by the learners. The fact that this subjective 
perception has no effect on learning perfonnance is an important lesson to learn - especially for the learner- 
oriented evaluation of texts. In future studies we will try to find more indices through empirical means, which 
may help us to gain a systematic understanding of the factors influencing the quality of learning texts. 
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Abstract 

The purpose of this paper is to describe a research done with Spanish universities, where professor’s 
practices at the virtual campus have been studied and the works dynamics proposed by students. The goal was 
to prove if there were usage patters or associated teaching models to different variables like: scientific field, 
platform usage time, within others. A qualitative methodology was used with 89 open interviews at 17 
universities. 

Key Words: Learning model, virtual environment, university teaching, didactic strategies. 

1. Introduction. 

While efforts have been devoted to the study of the quality indicators associated with technology and 
contents at e-learning environment, methodological aspects issues related with adaptation and harnessing or 
possibilities of e-learning, has been a field, comparatively underdeveloped. The absence of this methodological 
substrate that will include and rationalized practices developed in virtual environments of different learning 
models, makes it difficult to obtain predictable learning results, as well as to verify quality assurance needed at 
the teaching learning process 

Regardless of the methodology used, a transition is taking place from the conventional class in campus 
to cyberspace class. Professors and students act in differently in both types of classes, learning products are also 
different. Virtual environment courses and programs have emerged so quickly, that neither socially nor 
educationally philosophy has been developed, regarding the impact of these distribution methods. There is quite 
an amount of thought over the need to modify the teaching approach, but what is mostly used is to rehearse with 
traditional teaching methods in these settings. 

Failure is not in using ICT at a distance, nor in the sophisticated tools, but on the pedagogical design, 
more in the absence of this design (Spector, Wasson y Davidson, 1999; Cook, 2000). We use pedagogical, 
physiological and other related theories, but progress is needed in the theoretical principles for the design and 
implementation of virtual environments, since they are complex at the moment to incorporate organizational, 
administrative, educational and technological components (Avgeriou et al., 2003; Salinas, 2004; Garcia Aretio, 
2007). 
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The functions that a teacher must develop in these new learning scenes have been analyzed from 
different perspectives and in a continuous form in the last decade. The principal contributions from those 
studies are cantered in the focus change toward methodological update, necessary to obtain better results, in 
terms of learning. 

Literature is available regarding the analysis of technological and pedagogical possibilities that virtual 
tools offer us. One of the principal conclusions that emerge from these works is the assurance that ICT 
possibilities for education do not depend on technical potentiality of the tools used, like from the learning model 
used, from the way to understand and potentiate the relation between teachers and students and definitely, from 
the importance that teaching processes acquire in the virtual environments formation. Definitely, creating 
virtual effective teaching and learning environments, requires considering methodological changes at the 
deployed teaching techniques. 

If we understand teaching/learning like processes of knowledge creation that considers users 
characteristics, the organization and the technology, the teaching strategies divide one of the variables of greater 
richness to acquire quality teachings/learning results. Also within the set of possible strategies some will result 
more adequate in certain usage contexts. 

It seems adequate then to study how this process is being developed at university contexts, what types 
of teaching models are being used, what teaching and learning strategies and what learning activities. Having 
testable information makes it possible to identify and establish usage profiles from professors and teaching 
models. 

2. -Study discription 

The study presents the results from EA2007-0121 project: Teaching models in university virtual 
campuses. Methodological profiles generated by professors at the virtual environment teaching/learning 
process, project which was funded by the Directorate General of Universities of the Spanish Ministry of 
Education. 

2.1. Study objective 

In this study, professors usages of virtual campus and the work dynamics proposed to students were 
analyzed, in order to identify use patters or teaching models, related with different variables associated with 
professors (like scientific field, professional categories, teaching experience, age, time the teacher has been 
using the platform campus,...) 

With this approach and starting with the data obtained from professors, we hope to offer a general vision of 
VLE (Virtual teaching/learning environments) usage models and analysis elements from those, among the 
Spanish universities. 

2.2 Methodology 

The study must be framed between research methodologies for design and development, the ones that 
pretend to produce knowledge to better the educational design process, development and evaluation. In our case, 
we aim to better professors' teaching strategies. 

The study was done with a team of 27 researchers from different Spanish universities. 

Data gathering was done applying 89 open interviews at 17 different universities. The object of the interviews 
was to describe work dynamic and the subject organization that the professor thought at the virtual campus, 
attending virtual and on site processes activities. 

Data treatment was made applying qualitative procedure, to analyze interview contents 

2.3. Population and Sample 

Population was made up of all professors that developed activity at the virtual campus of the 17 
universities. 

From the total population a stratified sample of professors was taken, in relation to the knowledge field and to 
the time that they had been teaching at the virtual campus. Considering the capacity of the research team, 
original 100 interviews were planned. From 4 to 10 from each university, at the end 89 were actually applied. 
The professors sample interviewed show the general profile of an experienced professor, since majority (80%) 
have worked at the virtual environment for more than an academic year, they represent different knowledge 
areas of study, the typology of the subject thought is representative of the curricula at Spanish universities and a 
great deal. (48%) use Moodle like Learning Management System (LMS) or technological platform for teaching. 

2.4. Procedure 

Data gathering procedure and process include the following activities: 

• Design of an interview protocol. 

• Recording the interviews 

• Transcription of the sound files to text. 

• Qualitative treatment of the data by coding the interview transcriptions into significant units. 

• Category system construction. The building of the final category system was done after to. After 
reviewing and coding 25% of the interviews, a previously design category system, which was tested 
and review by experts to finish with the final construction of categories and codes 
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3. Results. 

Data analysis given by professors in the interviews let us observe differences regarding activities that 
the professor gives to students. The materials that are given to students and VLE usage to the management of 
the subject, through the calendar, blog, forum, or to make other activities like virtual tutorials. 

3.1. Tipology of virtual environment use. 

From the information given by professors at the interviews, a typology comes up, regarding the use of 
virtual environments, that are supported in other previous studies (Perez, Darder, Piccolotto y Salinas; 2006). 
Typology is organized with respect to the differences observed at “activities” and “materials” dimensions. 

We specifically identify 5 types of usage: 

• Type 1 professors who use the virtual environment to distribute materials, they might propose some 
kind of punctual activity in a voluntary way. 

• Type 2 professors who use the platform to distribute materials and who propose making individual 
obligatory activities. 

• Type 3 professors who use the platform to distribute materials and propose making individual activities 
and/or obligatory group activities. 

• Type 4 professors who use the platform to distribute materials and propose individual activities and/or 
obligatory group activities. This group of professors has specified in the interview that they do 
collaborative work also in an obligatory way. 

• Type 5 professors who use the platform to do individual, group or collaborative work activities. These 
activities are obligatory. They differ from the other profiles, that they do not distribute any kind of 
material. 

In order to see the differences, the typology is schematically presented in the table 1. 





Materials 


Activities 


Individual 


Group 


Collaborative 


TYPE 1 


X 


0 


O 


0 


TYPE 2 


X 


X 


0 


0 


TYPE 3 


X 


X 


X 


0 


TYPE 4 


X 


X 


X 


X 


TYPE 5 


0 


X 


X 


X 



Table 1. Types of virtual environments usage 



Distribution according to these types of use 

a. The assignment of identified types of virtual environment use, correspond in its majority to the first 
three types in 3, 1 and 2 order. The principal use of the platform include the materials presentation by 
students, with the activities proposal, with out the activities proposal for type 1 (28 cases) or with the 
activities proposal only individually for the type 2 (19 cases), or also group activities for the group 3 
(29) cases. In less proportion we observed that 8 correspond to group 4 and 5 to group 5 




■ TYPE 1 

■ TYPE 2 

■ TYPE 3 

■ TYPE 4 
TYPE 5 



Graph 1. Interview distribution according to the type of platform usage 

b. When relating the use and knowledge type of, there is certain domain of type 1 in Experimental Sciences and 
Technical Engineering, while type 3 denotes in Health Sciences and in Social and Judicial Sciences. In 
Humanities, no domain is clear even though the one with more frequencies is Type 2. 
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TYPE 5 

■ TYPE 4 

■ TYPE 3 

■ TYPE 2 
TYPE 1 



Graph 2. Interview professors distribution according to the type of platform usage and the general knowledge of 

the subject. 

c. The distribution of the types of uses in relation to the professors years of experience using the web to support 
teaching, no meaningful differences are shown with respect to its distribution at the different types 




TYPE 5 
TYPE 4 
TYPE 3 
TYPE 2 
TYPE 1 



1 year 



over 5 



Graph 3. Interview distribution according to type of usage in the platform vs years of experience. 



3.2 Professors profiles regarding methodological strategies use for web learning 

Observing at the same time, the level of on site and virtually in work dynamics and to the levels of 
frequency that stand out of the collected data in the categories, types of activities, used material and type of 
intervention that the teacher makes, general tendencies can be observed that sketch different professors' profiles 
In the following table we collect the principal elements and identifiers of each profile: 



216 



ON SITE 


100% face-to-face. The weight of the subject is on the classroom sessions. 
Platform is used to distribute material. 


COMPLEMENTARY 


Complementary use of face-to-face classes. The work dynamics is done at on 
classroom. The virtual environment is used to distribute the materials and 
some specific activity voluntarily and/or to give the activities that will be done 
in site. 


SUPERIMPOSED 


Superimposed use of the face-to-face classes. Even though the weight of the 
activity or dynamic is basically at on face-to-face sessions, it complements 
itself with materials distribution and doing activities at VLE. In this case no 
substitution or very little substitution is done of the time on classroom 
intervention at VLE, which could be considered some flexibility with respect 
to students that do not attend classroom sessions. 


ALTERNATE 


Between 50 and 70% of the subject; is done face-to-face and the rest virtually. 
In this case there is an explicit division between virtual and face-to-face 
procedure, that could be temporal or do some precise tasks. 


INTEGRATED 


Between 50 and 70% of activities are done face-to-face and the rest virtually. 
In this case there is not an explicit division between virtual dynamics and the 
face-to-face that could be temporal or for some specific tasks. 


VIRTUAL 


professors give their class completely on line 



After studying these general profiles described, more definite profiles appear, that refer to types of 
activities, evaluation, tutoring, management, of the environment by the professor, they are presented as 
alternatives to what is happening at VLE in Spanish universities and they can contribute to the study of 
methodological elements of the above profiles. 

The following data relates to the type of use and professor's profile in VLE. These profiles will be hard 
to tell apart, because more than a classification it is an spectrum that includes from the totally on site profile that 
only offers materials (like presentations, tutorials and notes) up to the one working completely virtually, bases 
all its intervention in activities with out offering materials in the environment. 

64% of professors interview belong to 5 of the profiles defined. Higher frequencies are decanted by 
teaching methods are decanted taken in consideration. By one face-to-face profile with the ICT help (on site, 
complementary and superimposed) When distribution is done between in site and virtual is more balanced, an 
integrated profiles stands out, without explicit division between on site and virtual and dynamic group work. 





Type 1 


Type 2 


Type 3 


Type 4 


Type 5 




On site 




0 


0 


0 


0 


16 


Complementary 


12 


3 


0 


0 


0 


15 


Superimposed 


o 


v 


13 


1 


1 


22 


Alternate 


0 


2 


3 


2 


2 


9 


Integrated 


0 


4 


9 


3 


0 


16 


Virtual 


0 


3 


4 


2 


2 


11 




28 


19 


29 


8 


5 


89 



< 5 % (between 5 and 1 6 prof) 




2 - 5 % (3 a 5 prof) 




> 2 % ( 1 o 2 prof) 




Without professors 





In all cases, profiles that present less frequency, are still interesting., when they include valuable experiences in 
relation to the methodology used and the work dynamic developed 

If we establish the diagonal that runs in the graph from left comer at the top, towards the inferior to the right we 
can see an evolution in professors practices in VLE which let us show that common practices are based in 
certain methodologies centered in the teacher even though we find valuable practices that develop 
methodologies centered in the student. From this last one, we find examples in the dynamic collaborative 
profile. 
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4.. Evaluating results obtained 

The usage of ICT at Spanish universities could be considered something of general use. The evolution 
of the observed phenomenon the previous studies (CRUE, 1997, 2004; Area, 2001; Valverde, 2004; Sangra, 
2004; Infante, 2005; etc..) and the results obtained here verify them. This utilization presents, according to the 
results obtained, the following characteristics; 

• The general usage of VLE is distributes among the first types that suggest the classification proposed 
by Roberts, Romm and Jones (2000). 

• 65% of professor show that they keep on with face-to-face teaching but we observe the profiles 
described in the study, the first three profiles, on site, complementary and superimposed, which its 
presence if practically 100%, they represent 60% of the interview professors. 

• The course model used at the studied universities matches the original model, the one that shows a 
minimal use of the platform and is centered on putting materials at student's disposition, with one 
model characterized by the presence of basic teaching materials, complementary and of the 
organization, together with a activity individual and group proposals. The majority of activities that are 
asked for are evaluated and affect the final grade of the subject 

• That is why and keeping in mind that a high percentage of professors consider it necessary the access 
to the platform in order to keep up with the dynamism of the worked planned for the subject (to consult 
the material, do activities and/or hand them in). We can say that the use of virtual learning 
environments exceeds the support to face-to-face sessions, and supposes one proposed extension of the 
same drawing the web's potential to blend time and spaces. 

• Subject typology developed in VLE reproduce the actual configuration university curricula, so we can 
conclude that cannot be appreciated differences related to matter in VLE 

• The presence of teaching material stands out. If we keep to basic materials, 79% of teachers manifest 
they usually or always presentations in classroom. 77% of the practice guides and the proposed drills 
and 72% of study guides. On the other hand, 94% of interview students the use basic classroom 
material. Either in study guides and instruments for practices or the form of developed contents. 
Parallel a relation is shown between the used profile and the variety of materials that are the service of 
the student, participating in multimedia approaches 

• In the same way, an evolving tendency of different profiles cannot be appreciated, related with the time 
the teacher's experience at the virtual campus 

• Relating to defined general profiles, we can say that after studying in relation to other variables, not 
only type, environment and experience that we have presented here, they are presented as 
representative of what is happening in VLE of the universities and it could contribute to the study of 
the methodological aspects, to contemplate in the same. 

• To define more detailed profiles, we can start from the relation between general profiles and the 
definite use types- Si we look the table we can find 18 possible profiles more detailed related to these 
variables, use type and general profile associated to others like the type of the materials offered, types 
of activities developed or the evaluation used. In any case, these profiles will be very difficult to tell 
apart, so more than one classification, what they can take is an spectrum, that includes from the profile 
completely on site that offers only notes material type, until the one that working totally in virtual form 
bases all intervention in activities, without offering materials in the environment 

5. Conclusion 

Analyzing the results obtained, lets us think that starting with the same, a reference schema could be 
organized, a reference schema in relation to the pedagogy practices developed in teaching/learning virtual 
environments, that could serve for the analysis and the methodological proposals and resources aimed to the net 
of upper studies formation and as base for the confection of a planning and development guide for planning and 
development of training and coaching activities. 

Gathering, systematization and diffusion of the most usual methodological Strategies at universities virtual 
environments, contributes, on one side, to go ahead on the status of the courses resources offered digitally and 
to be able to know what today’s reality is. On the other side, it offers the possibility to compared the real 
experiences that the professors are using for pedagogy and finally to give a relation of resources and experiences 
that professors or our university could use in the redesign of the same experiences 
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ABSTRACT 

We are presenting the results of a study carried out on methodologies used in the virtual university campus. 

The study was carried out using an online questionnaire, which was answered by 782 professors from 17 
different Spanish universities. 

The putpose of the questionnaire was to describe teachers use of the virtual learning environmet and the 
working dynamics they propose to their students, as well as to verify if patterns were given or teaching models 
were related to different variables 

KEYWORDS 

Teaching models, didactic strategies, learning virtual environment, e-learning. 

Introduction 

In current times, it is expected from the university to be the leader at innovation and research system. 
This supposes a present and future planning regarding e-learning possibilities and an adequate use of ICT in 
teaching, in line with the main trends in the universities at an international level. It is know that Spanish 
universities are incorporating ICT and as consequence, they have started to build virtual learning environments 
for teaching and learning (VLE) making it possible network connections to a larger number teachers and 
students in training activities either at a distance since they complement the conventional classroom and they 
characterizes above all for a major flexibility. 

We start from the virtual learning environment conception, like a space or community organized with 
the purpose to obtain learning, and for it to take place, it requires certain components: a pedagogical function 
(that makes reference to learning activities, to teaching instances, to learning materials, the support and tutoring 
in stock, to the evaluation, etc) the appropriate technology (that makes reference to the tools selected in 
connection with the pedagogical model) and the organization framework (that includes the space, organization, 
the calendar, the community management, etc.. But also the international framework and the implementation 
strategy) (Salinas 2004). 

The application to institutional and corporate e-learning contexts has centered its efforts up to now in 
technology (LMS platforms, content treatment, communication and collaboration spaces etc) and in its contents 
and materials. But, e-Learning understood from both points of view , centered in technology and centered in 
information, has neither fulfill the promises of success that it seem to offer, nor has explain successfully how 
they develop learning activities in a virtual environment. The failure is not on the use of distance ICT nor in 
how sophisticated tools are, but the pedagogical design or in its absence better said. (Spector, Wasson and 
Davidson, 1999, Cook 2000). It is common to insist that quality resides in contents, but to make effective virtual 
environments for teaching and learning that obtain results in terms knowledge construction, requires considering 
methodologies changes and didactic strategies changes implemented (Salinas, 2004) 

Characteristics determination in terms of effectiveness, that teaching strategies must include is very 
complex process, since great amount of variables interact, conditioned by each of the different contexts in which 
they develop. 
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It seems adequate, to study how this process is developing to the interior of our universities and what 
knowledge we can get out of it, having testable information that permits us to identify and establish certain 
usage of virtual environments teacher profiles, didactic models display by teachers in its daily practice. 

Attention to this phenomenon has some trajectory at Spanish universities and the new modalities of teaching in 
VLE and institutional projects to exploit them, has been reached from different perspective, since ICT teaching 
incorporation produces a great processes diversity: 

• Communication and distribution system for teaching material, adequate computer mediated 
communication working in virtual environments, etc. Related both with process selection and platform 
and software exploitation, tools, the analysis of the technological and pedagogical possibilities that the 
developed tools offer us. 

• Rol changes from professor as well as from students, the first one passing from actor to situation and 
learning mediated environments designer and the second from spectator to actor of its own learning. 

• Innovative processes associated with the incorporation of the university to the European higher 
education space 

• Professors methodology changes that imply the use of new e-learning techniques, to complement 
students formation. 

From all of these changes, may be the most important ones from quality point of view, are the methodological 
changes, since they incorporate elements from all the rest. 

The research presents part of the results from one of these studies we developed as EA2007 -0121 . 
Teaching models at virtual university campuses. Methodological patters generated by professors at 
teaching/learning processes at virtual environments. On line questionnaire constitutes the first of three phases 
of the study. Which was done by a team of 27 researchers and data collected belong to 17 Spanish universities. 
The research questionnaire was answered by 782 professors aiming to study the virtual campus usage by 
university professors and the work dynamics they propose to their students, with the goal to find out if there are 
any patters of use or didactic models related to different variables like: scientific environment, time the 
professor has been using the platform at the campus or the institutional of ICT integration to the one it belongs 
to., etc. 

Study objectives. 

The principal objective is to detect patters of use or usages of certain teaching models in the virtual 
campus, related with different variables: scientific field, professional categories, teaching experience, age, time 
the teacher has been using the platform campus, institutional ICT integration program in which he participates, 
experiences in ECTS, etc. 

This objective could be detailed as follows 

• To identify use profiles existence of different platforms at virtual learning environments, that could 
lead to the organize different teaching models to use. 

• To analyze if this virtual learning environment use by professors, responds to some general parameters: 
knowledge field, ECTS projects incorporation pertinence, teaching innovation projects, experience at 
the virtual campus, age etc. 

• Create a reference schema with the results obtained, that could serve as a guide and reference for the 
planning and development of learning and tutoring activities, that could give methodologies and 
resources for decisions making over methodological strategies, to utilize at learning networks in higher 
education. 

Research procedure description 

The research procedure consisted of an on line survey to a sample of professors at the universities 
participating in the study (17) The objective was to identify usage profiles with respect to different variables. 
All these through: 

Adapting data gathering instrument (Questionnaire, design of the one line form, validating the 

questionnaire.) 

Applying the questionnaire 

Data capturing 

Review, classification and debugging of the results obtained. 

The research universe was made up of all the professors that developed activities in the virtual campus of each 
university. We received 782 responds. The distribution by universities is represented in Fig. 1 . 

The questionnaire contains the following elements: 

• General data, age, number of years in the virtual campus, etc. 

• Type of activity; lecture, reflections, analysis, creating, observation and findings. - 

• Grouping 

• Learning activities and teaching techniques. 

• Learning materials 
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• Evaluation. 




Results 

Within the most significant results obtained once data was treated, we can point out: 
a. Years of on line teaching experience. 

From professors that answered the questionnaire 12% has one year or less of on line teaching 
experience, 35% between 2 and 3 years, 30% between 2 and 3 and 20% more than 5 years (Fig 2). 




□ 1 year or less ■ between 2 and 3 □ between 4 or 5 □ more than 5 □ blank 



Fig 2 



Years of on line teaching experience (in percentiles) 

On the other hand, according to the knowledge field to which the subject being thought (see fig 3) 
belonged, the majority belonged to social and judicial sciences (41%) In the rest of the area, its distribution is 
very similar. 
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Fig 3 Percentage of professors according to the area to which the subjet belonged, 
b.- Teaching Modality 

Teaching modality that professor manifest, participating in virtual campus project at virtual 
universities, is still face-to-face (65%) blended modality (16%) and distance (19%) do not reach a significance 
proportion. 




■ distance 

■ face to face 

■ blended 



65% 



Fig 4 Modality teaching. 

Regarding platform usage, 80% manifest they have used some, with respect to 20% that doesn’t use or 
doesn’t answer the question. The platforms mostly used are WebCT/Blackboard and Moodle, from which 36% 
and 35% of professors respectively, affirmed they have used them. Regarding the use of other platforms like 
Dokeos, SUMA, SAKAI or the belonging to UOC, the usage percentages for each of them are around 1% 
(which is equivalent to 7 and 15 professors). 
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2% 



18% 




80% 



■ no 

■ yes 

■ blank 



Fig 5 Professors percentage that use some platform for teaching, 
c.- Learning activities 

Typology of proposed learning activities in the questionnaire in order to show the frequency while 
being done, the lecture is the most generalized, together with reflexion and analysis. The creation and discovery 
in that order, are the less utilized. . 

100 % 

90 % 

80 % 

70 % 

60 % 

50 % 

40 % 

30 % 

20 % 

10 % 

0 % 

■ never iafew ■ sometimes ■ often ■ always 




Fig. 6 Use frequencies according to type activities (in percentiles) 

If we compare use frequencies of different types of activities with knowledge area to which subjects , 
being though by professors, belong to, we observe, just as fig. 7 shows, the type of activities that are mostly 
proposed by area they are also, reading, reflex ion and analysis. 
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ig. 7 Use frequency of activity types by knowledge area, 
d. Learning activities and teaching techniques 

From the different techniques and activities from the questionnaire, the ones that are mostly used in a 
greater proportion by professors (78%) mostly or always are group activities and simulating. It is correct to say 
that demonstration, symposium and study cases show less professor proportion, (57%, 58% and 56% 
respectively), with respect to the two last ones, the ones that manifested using it sometimes are 37% and 36% 
respectively. 



Projects 
Problems 
Case Studies 
Group activitites 
Social Interaction 
Role Play 
Simulations 
Symposium 
Discussion 
Demostration 
ConsultExpert consultation 
Presentaciones online 
Search 
Materials 

0% 25% 50% 75% 1 00% 




■ never ■ a few ■ sometimes ■ often S3 always 



Fig. 8. Usage frequency for learning activities and teaching techniques 
Considering techniques and activities that according to professors they use often or always together 
with the experience with online teaching, we observe as Fig 9 shows, a general level techniques less utilized 
seem to coincide with professors that have two or more on line teaching experience (they relate to case studies, 
symposium and demonstrations). But professors who have a year or less of experience the techniques used in 
lower proportion are: case studies, materials access and project methode. 
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Materials 



Searching 
Online Presentations 
Consult Expert 
Demostrations 
Discussion 
Symposium 
Simulations 
Role Play 
Social Interaction 
Group activities 
Case Study 
Problems 
Projects 




% 



■ > 5 years ■ between 4 and 5 ■ between 2 and 3 ■ < 1 year 



100 



Fig 9 Teaching techniques and professors experience. 

With respect to most commonly used techniques by professor with more experience, the results 
obtained show that role play, problem resolution and debate in that order. Professors with 2 to 3 years of 
experience use simulations in first place and problem resolution, group activities and debates in equal 
proportion. The ones that have between 4 and 5 years of experience , use in first place simulations followed by 
social space and group activities as well as online presentations. Professors with 5 years or more manifest the 
use in higher proportion the social spaces (80%), followed by simulations and debates. 

Analyzing professors that manifest to often use or always use different learning activities and teaching 
techniques according to the knowledge field to which the subject they teach, belongs to, we can see differences 
between the used techniques (Fig 10) In health sciences the technique mostly used by professors is 
simulation(88% manifest has used it often or always) followed by the resolution of problems and debates, 
meanwhile the less utilized are, first symposium, followed by demonstrations, study cases and project work. In 
experimental sciences area the technique mostly used was social space followed by problem resolution, debate 
and role play, and the least used were expert consultation, search and information retrieval, the access to 
materials and demonstrations. But on social and judicial sciences they manifest to use more simulations, on line 
presentations and problem solving and less study cases, symposium and Project work and demonstrations. In 
teaching techniques, as more used technique stands out debates, followed by group activities, simulations and 
search information and the least used, symposium, Project work, and demonstration. To finish, in Humanities 
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the mostly used was expert consultation, followed by group activities, simulation and social spaces and the least 
used, study cases, and demonstrations. 




Health S. 

Experimental S. 

Social and Judicial 

Technological Eng. 
Humanities 



Fig 10 Teaching techniques and knowledge area 



e. Types of learning materials and resources used. 

The types of teaching materials and resources used by professors has been analyzed, globally and 
separately following the classification showed in table 1 .- 

Globally we observe that the mostly used materials are the ones in basic categories and academic 
guidelines. So 79% of the professors manifest to often or always use presentations in class, 77% practice guides 
and practice proposals. 72% study guides. Around 75% of the professors confirm they have used interactive 
multimedia materials, virtual labs, discussion forums or glossary. 

If we take into consideration the variable “sometimes” other types of resources are used for learning, 
like practice and exercise materials, on line applications, tutorials, class notes or schemas, or the subjects 
syllabus. 

The type of material less often used by professors are, first, video files, followed by on line interests groups, 
animations, Reading articles and on line magazines. 

Regarding basic materials the ones that are often or always used in higher proportion are: class 
presentations, followed by tutorials and class notes or schemas. 

Regarding the materials types that we have classified like activities, the ones that are mostly used in 
proportion are practice guides and practice proposals. The ones that are least ones used are auto evaluation 
exercises and previous exams. 

The proportion of professors that manifest using study guides (72%) often or always is higher tan the 
program utilization (53%), if the ones that sometimes utilize the program, it will be the material classified as 
academic guides more utilized (by 90% of the professors) 

From the different formats used, conceptual maps are the one mostly used or always in a less proportion but 
they are the ones that a higher number manifest using them often. The same happens with images or drawings 
and simulations. 
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BASICS 


• Tutorials 

• Classroom presentation 

• Notes, schems 


ACTIVITIES 


• Practice proposals 

• Selfevaluation exercises 

• Previous exams 

• Glossary 

• Materials for exercices/Practice 

• Electronica Diary or weblog 


ACADEMIC GUIDES 


• Syllabus 

• Study guide 


FORMAT 


• Video files 

• Sound files 

• Animations 

• Images, drawings, photos 

• Interactive multimedia material 

• Concept maps 

• Virtual labs 

• Interest groups 

• Applications on line 

• Reading articles 

• On line datebase 

• On line magazines 

• Simulations 

• Electronic books 


OTHERS 


• Forum translation 

• Discussion forum 



Table 1. types of materials 

The ones mostly used are in first place, interactive multimedia and virtual labs, followed by on line 
data bases and magazines on line. If we point out that taking into consideration professors proportion that utilize 
little or nothing, as a result we have that these last ones are the one that are less used together with reading 
articles, interests groups, animations and videos. 

Taking into consideration, the ones that manifest to use it sometimes, the mostly used formats are: 
sound files, online applications, followed by online data bases and animations. 

To finish, with respect to the types of materials included in “other” category, discussion forums are the ones 
most often used or always, followed by the forum transcriptions and students work. If we take in consideration 
occasional usage the three types of materials are used by professors in a similar proportion. 

CONCLUSIONS 

Of the results obtained through this study, the main conclusions are: 

• The use of the virtual environment greatly overcomes the face-to-face sessions; this implies their wider use 
taking advantage of the network potential in order to make space and time more flexible. 

• It is possible to teach within the EVEA any sort of subject no matter their characteristics and knowledge 
field. 

• The use of didactic material is outstanding 

• There is a relationship between the use and the variety of materials at students disposition , creating a 
multimedia approach. 

• The dominant teaching pattern in our universities is the one that overwhelmes elements and strategies of the 
traditional teaching model even though we are starting to observe teachers that apply didactic strategies 
based on student 

• The most used didactic techniques are the teaching material access, the search and information retrieval. 
Other techniques have appeared showing an evolution towards student’s based methodologies. 
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• As good relationship between the profile usage and the variety of teaching materials at the student’s 
disposal has been shown. 

• The organizational patterns need to be separated into different levels: those that affect the institutions and 
those that affect the teacher’s tasks. 

• The organizational model should be defined based upon two-dimensional criteria. It would be better to keep 
in mind arguments such as the transaction distance or the combination of two or more of these dimensions. 
We need to invest in new typologies and it would be convenient to look for agreements regarding the main 
dimensions that could be used as central idea for these typologies. 
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The Effects of Integrating a Virtual Lab in an Online Class: Student 
Perceptions and Implications for Future Practice 

Bude Su 

Abstract 

Online instructors are still facing a number of challenges despite the advancements in today’s technologies. 
One of the challenges of online teaching is providing visible guidance within a virtual learning environment. Visible 
guidance becomes especially important for those courses that involve lab sessions. The focus of this study is to 
empirically investigate what and how different instructional strategies —especially the use of a virtual lab — are used 
to support student learning. Two classes of sixty graduate students participated in the study. The same content is 
delivered using different instructional strategies and different delivery methods. The course evaluations of these two 
groups are compared and findings are discussed. 

Introduction 

Activities that require student interaction and idea sharing are known to promote reflective thinking. Since 
students are physically isolated in online learning environments, any decision-making or exchanging ideas requires 
more time and efforts compared to those in face-to-face settings. To help facilitate student needs, both instructors 
and students must adjust themselves to the delivery method and use of new technologies. 

The types of technologies and how they are used can impact both the quality and depth of learning process 
and learning outcomes. How technologies facilitate personal connections with instructors and students affects the 
degree of social presence among them (Duarte & Snyder, 1999). Previous research has demonstrated the use of 
active instructional strategies for improved learning outcomes in science education (Bredderman, 1982). Obviously, 
watching a PowerPoint lecture online is not an active way of learning certain content. Such passive learning 
becomes especially problematic when it comes to those courses that require a lab component. Many agree that 
online learning environment certainly brings challenges, but also provides opportunities for innovative instructional 
practices. Incorporating a virtual lab into an online class is one of the relatively new approaches in online 
educational practice. Compared to a print-based job-aid on how to carry out lab activities, a virtual lab can facilitate 
a more engaging learning experience and helps decrease instructor workload on answering lab-related questions. 
Therefore, in theory, such active instructional strategies enhance learning outcomes and increase student 
satisfactions. But is this true in practice? 

In order to understand how certain instructional strategies such as using a virtual lab may affect student 
learning and satisfaction, the researcher conducted a two-semester exploratory study to examine: 

1 . Is there a statistically significant difference in the willingness to attend lab and the perceived 
usefulness of the lab between two groups (virtual lab group vs. face-to-face lab group)? 

2. How do students perceive the virtual lab or the face-to-face lab in terms of helping them to understand 
statistical concepts and procedures? 

3. What factors are perceived to make either type of lab most effective and why? 

Study Overview 

The immediate purpose of this study is to investigate student perceptions about their experiences studying a 
graduate level course, “Assessment and Evaluation”, using different learning strategies. In this course, students are 
asked to explore and develop methods of assessing learning and performance through the use of psychometric 
techniques. The first four weeks of the classes focus on reviewing statistical concepts and procedures. Learning 
statistics online is considered to be very difficult. Some even claim that it is impossible to do it effectively. Two 
classes totaling 60 students participated in this study in two different semesters. The same content was taught to the 
participants using different pedagogies. One class of 27 students received text-based instructions with a weekly face- 
to-face optional lab while the other class of 33 students had a weekly virtual lab along with the text-based 
instructions. Participant age ranged from 24 to 65. Most of them chose this blended learning program due to work 
and family commitments. Over 90% of the participants are employed fulltime. The virtual lab was done through 
Adobe Connect where the instructor could share her computer screen with the lab attendees while explaining things 
through audio. The students could ask questions via a text chat window or using a microphone. The multi-channel 
audio conversation feature of Adobe Connect was somewhat problematic, such as hearing back own voice echo. 
Therefore, one-way audio delivery was used (only the instructor talking) with the students typing their questions into 
their text chat boxes. 

An end-of-course survey was used to collect data and to explore issues related to perceived effectiveness of 
the virtual lab used in the online graduate assessment and evaluation course. Survey data was collected using the 
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course management system that allows anonymous responses. The survey consisted of 15 questions that focused on 
student perceptions of effectiveness of the instructional strategies and delivery methods used in the class, in terms of 
increasing their understanding of the concepts covered in the class. 

Results 

A total number of 38 students responded to the course survey. Twenty-one of the respondents were enrolled 
in the first term where an optional face-to-face lab was offered along with text-based instructions. Seventeen of the 
survey respondents were enrolled in the second term where a weekly virtual lab was offered along with text-based 
instructions. Students in the both groups thought that having a lab for the statistics components was helpful. 
Although there is no statistically significant difference in their responses toward the statement “the weekly lab was 
helpful”, the students who enrolled in the virtual lab class rated the usefulness of the lab higher than the group who 
had optional face-to-face lab. 

Table 1: Perceptions on Lab Usefulness and Intention of Attendance 





Virtual lab 


Face-to-Face Lab 


Weekly lab was helpful 


4.6 


4.1 


I would come to lab sessions 
regularly 


4.5 


3.3 



While the idea of face-to-face lab was attractive, not all of them were willing to drive to campus and attend 
the lab. The rating for “I would like to come to the lab session regularly” resulted a statistically significant result 
between two groups (two tailed t=2.03, p<.01). The face-to-face lab group received 3.3 on average out of 5 possible 
points. Student comments provided explanation for the low rating. 

“It is an online course and I would not have time to attend a weekly lab [on-campus].” 

“As an out-of-area student, I would like the lab - but would prefer it be done online.” 

Clearly, a face-to-face lab was not an ideal option for the given target population and needed another kind 
of intervention. That was why the instructor modified the lab delivery mode from face-to-face to a virtual lab when 
the course was re-offered the second time. Despite of the technical issues at the beginning, the students who were 
enrolled in the virtual lab class highly rated the effectiveness of the virtual lab (4.6 out of 5). Some of the comments 
were: 

“Actually the virtual lab was very helpful. Participation in the lab gave the atmosphere of a real classroom. 
Also, gave chance to everybody to interact more than just emailing the professor our questions.” 

“Even though I wasn’t able to attend the lab, I was really happy that the lab sessions were recorded, so I 
could listen to them at my own pace.” 

“I attended the lab and used the recorded session for the one time I missed the lab. It was helpful to prepare 
my questions in advance and I learned a lot from other students’ questions.” 

Needless-to-say, the students appreciated the virtual lab component of the class and they were happy that 
the recording was available for them to access after the lab session. This is another good feature of Adobe Connect, 
which allows you to record the entire lab for future use. 

One interesting perceptual difference between the two groups occurred on answering the question, “if there 
is a virtual office hour, which communication methods do you prefer”. 

Table 2: Preference Differences in Communication Methods 





Virtual Lab Group 


Face-to-Face Lab Group 


Text chat 


2.7 


3.9 


Telephone 


2.9 


3.4 


Video conferencing such as 
AIM/iChat/Skype, etc. 


4.5 


3.6 



The group who had experience using video conferencing such as the Adobe Connect wanted to use it much 
more than the other group who did not have experience using it (t=2.47, p<.01). On the other hand, the experienced 
group did not want to go back for text chat (t=-2.93, p<.01). Like the old saying goes, “You don’t know until you 
try”. This result seems to indicate that those who did not experience using media-rich two-way communications had 
no clue how much better it could be than using simple text chat. 

One of the challenges of blended learning is the balancing of the degree of blending. To understand the 
important factors that adult students perceived as important, the researcher asked the students to rate a few items. 
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The result showed that the students perceived the flexibility to learn at their own pace as the most important factor 
for their successful learning (Mean=4.3). The second important factor was the depth of reflection, where the students 
had enough time to thoroughly digest learning materials so they could generate deeper and more thoughtful 
reflections (Mean=4.3). Spontaneity that allowed for real time conversations and discussions was rated as 
important, but was viewed as not critical to learning success in general (Mean=3.8). Face-to-face human connection 
was rated as the least important factor among the four factors listed in this section (Mean=3.4). 

The factors mentioned above tend to provide a reasonable explanation as to why the virtual lab was more 
successful than its face-to-face option. First, it met the criterion of flexibility. The students were able to participate 
in the lab session as long as they had an Internet connection, no matter where they were. They did not need to travel 
to the school as the other group did. Second, the recording feature of Adobe Connect provided not only another level 
of flexibility, but also a chance for them to digest on their own later. If this feature was unavailable, I do not think 
that we were able to see the statistically significant differences on several results between the two groups. Third, the 
virtual lab allowed for a real-time conversation and discussion, where the students were able to hear and learn from 
other students’ questions. Because it satisfied all three of the features that the students perceived to be very 
important for successful learning, the virtual lab component of the class was highly rated by the students. And of 
course, this positive attitude, in turn, resulted higher learning outcomes and student satisfaction at the end. 

Implications 

Learning from personal experience through trial and error, I would like to provide a few suggestions for the 
instructors who have similar needs in the future. 

Be brave and try new technologies. You don’t know what you are missing until you try! I was hesitant to 
use Adobe Connect in the beginning because I was already overwhelmed with all the courses and other tasks in my 
job. However, I am glad I did use it because it made my teaching so much easier during the semester. Challenge and 
opportunity always go together, and taking the challenge is well worth for this situation. 

Prioritize student needs and try to meet the critical needs first. When I offered an optional face-to-face lab 
for my online students, I thought that most of my students could come because they are local. But it was difficult for 
the students to come to the lab not only because the lab was offered at a fixed time, but also because they were not 
used to the idea of meeting in-person regularly in this online program. Flexibility (both in when and where) is the #1 
need for these students and you have to satisfy this need before going any further. 

Let the students write down their questions before the virtual lab session. Since the students needed to type 
in their questions, it was more effective if they prepared the questions ahead of time and then cut and pasted them 
into the text block during the lab. This also helped to decrease any possible embarrassment from typing and spelling 
errors. 

Build a friendly communication atmosphere. It helps a lot if the virtual lab provides a friendly 
communication atmosphere. This can be achieved through many tactics such as encouraging students to ask 
questions, providing positive feedback to their questions, acknowledging your limitations of answering certain 
questions, etc. Online students do not see each other often so they value the virtual lab session very much. 
Conclusion 

Results of this evaluative research show that online students do not like the optional face-to-face lab 
because it is unrealistic to have an on-campus lab for an online class. Not everyone lives within driving distance and 
most of the students have a full-time job. As a result of not being able to fully utilize the lab, they struggled with the 
statistics part of the class in general. On the other hand, the students who had virtual lab option highly rated the 
value of the lab not only because their questions were answered, but also because they learned a lot from each 
other’s questions. The recording feature of the lab was very helpful as well. 
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Effects of Students’ Identity Salience on Their Attitudes toward Face-to-face 
Classroom Collaborative Learning Activities 

Feihong Wang 

John K Burton 

Abstract 

This study explored the effects of students’ identity salience on their attitudes toward face-to-face 
collaborative learning activities. In addition, this study also sought to find out whether there was an interrelationship 
between students’ identity salience and their ages and nationalities, as well as whether students’ prior experiences of 
collaborative learning affected students’ attitudes toward collaborative learning? Twenty graduate students attended 
the study by completing a survey. The findings indicated that students’ attitudes toward the collaborative learning 
approach were affected both by students’ identity salience and their prior collaborative learning experience. 

However there was no significant relationship between students’ identity salience and their age ranges and 
nationalities. 

Introduction 

In educational settings, the issue of peer collaborative learning is often the subject of the studies (Ozmantar, 
2005). In peer collaborative learning, students with similar levels of competence share their ideas in order to jointly 
solve a challenging learning task that cannot be accomplished on their own prior to the collaborative engagement 
(Damon & Phelps, 1989; Goos, Galbraith, & Renshaw, 2002). The peer collaborative learning environment provides 
students with a free and open forum to facilitate active and productive exchange of ideas (Driscoll, 2000). Studies 
have found that the collaborative learning approach could possibly increase learners’ motivation (Bruffee, 1999), 
catalyze the development of critical thinking and problem-solving skills (Waite & Davis, 2006), build social 
atmosphere for small interdependent groups to con-construct knowledge (Vygotsky, 1976) and so on. However, not 
all forms of peer interaction are considered collaborative (Ozmantar, 2005). As Thomson and Perry ( 1998) pointed 
out “collaboration is like cottage cheese. It occasionally smells bad and separates easily” (p.409). The failure of a 
collaboration may be attributed to problems such as communication, effort-avoidance (Salomon & Globerson, 

1989), underlying paradoxical tensions (Smith, 2005), and perceived group statues (Barron, 2003; Nuthall, 1999). 
Researchers claimed that the effectiveness of collaboration only happened under certain circumstances (Dillenbourg, 
Baker, Blaye, & O'Malley, 1996; Slavin, 1999). Three distinctive and interrelated conditions for effective 
collaboration are group members’ mutuality, interdependence (Ozmantar, 2005), and equality (Damon & Phelps, 
1989). Although researchers have pointed out these conditions for effective collaboration, the fundamental causes of 
collaboration problems or motivational factors for the mutual communication, continuous sharing, and perspective 
taking have left mainly untouched. 

An individual’s identity has three aspects: personal, relational, and collective identity (Brewer, 1991; J. E. 
Cote & Levin, 2002; Y. Kashima, et al., 1995; Lord, Brown, & Freiberg, 1999; Sedikides & Brewer, 2001). Only 
one identity aspect takes the dominant position for a person in a given situation (Burke, 2003; Lord, et al., 1999; 
Stryker, 1968; Stryker & Burke, 2000). Thus, it makes three different types of identity salience; individual, 
relational, and collective identity salience (Brewer & Gardner, 1996; E. S. Kashima & Hardie, 2000; Sluss & 
Ashforth, 2007). Individual’s identities, in general, have potential impacts on motivation, judgment, self-esteem, and 
behaviors (Breckler & Greenwald, 1986; Brewer & Gardner, 1996; E. S. Kashima & Hardie, 2000; Triandis, 1989), 
so does an individual’s identity salience (Benson & Mekolichick, 2007). Among the three identity salience, one type 
of identity salience may be more consistent with the collaborative learning atmosphere than the other. In other 
words, students with one type of identity salience may have more positive attitude toward collaborative learning 
than the other. This study used identity salience as the prism to explore students’ attitude differences toward 
face-to-face collaborative learning. In addition, this study also looked into the relationship between students’ prior 
collaborative experience and their attitude toward collaborative learning; as well as the relationship between 
students’ identity salience and their age ranges and nationalities. 
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Review of Literature 



Collaborative Learning 

Collaboration with others has always been a central form of human activity (Barron, 2000). Collaboration 
has started to gain its attention and prominence in the educational field since the last two decades of the 20 th century 
(Dornyei, 1997; Martin, 2007). In the educational realm, collaboration is specified as collaborative learning, that is, 
as defined by Johnson and Johnson (1996), “the instructional use of small groups so that students work together to 
maximize their own and each other’s learning” (p. 786). Collaborative learning evolved from the work of some 
psychologists such as David Johnson, Roger Johnson, and Robert Slavin (Alavi, 1994). Collaborative learning was 
claimed to be a complex and not universally defined concept (Resta & Laferriere, 2007), and the meaning of it was 
defined or adopted differently by researchers based on their specific perspectives. However, the fundamental 
elements of collaborative learning have been kept the same: group based activities, shared academic goals, mutual 
help, and resource sharing. 

Collaborative learning and cooperative learning 

Collaborative learning and cooperative learning have been used interchangeably by lots of researchers 
(Aiken, Bessagnet, & Israel, 2005; Sawyer, 2006). Even definitions of collaborative learning and cooperative 
learning are sometimes hard to distinguish. Kirschner (2001) summarized seven shared common elements between 
collaborative and cooperative learning: 1) learning is active; 2) the teacher is usually more a facilitator than a “sage 
on the stage”; 3) teaching and learning are shared experiences between teacher and students; 4) students participate 
in small-group activities; 5) students take responsibility for learning; 6) students reflect on their own assumptions 
and thought processes through discussing and articulating ideas in a small-group setting; 7) social and team skills are 
developed through the give-and-take of consensus-building. 

Some researchers insisted on the necessity of differentiating collaborative learning from cooperative 
learning. There are three main differences between these two learning approaches. The fundamental difference is the 
structure ( Dillenbourg, 1999; D. W. Johnson & Johnson, 1996; Slavin, 1995, 1997). Cooperative learning requires 
the division of labor and role taking among participants and the task is split into independent subtasks. Collaborative 
learning, as John and Johnson (1996) point out, is historically less structured and more student directed than 
cooperative learning. Collaborative learning provides only vague directions to teachers about its use. In a 
cooperative learning activity, each participant is required to be responsible for a portion of the problem solving, 
whereas collaborative learning involves the mutual engagement of participants in a coordinated effort to solve 
problems together (Dillenbourg, 1999; Lehtinen, Hakkarainen, Lipponen, Rahikainen, & Muukkonen, 1999; 
Roschelle, 1996; Teasley, 1995). Consequently, cooperative learning is normally associated with well-structured 
knowledge domains, while collaborative learning is associated with ill-structured knowledge domains (Slavin, 

1997). Research on cooperative learning tended to emphasize the importance of individual accountability (D. W. 
Johnson & Johnson, 1986, 1993; D. W. Johnson, Johnson, & Scott, 1978; R. T. Johnson & Johnson, 1983; Slavin, 
1999), while research on collaborative learning focused more on the group as a whole. Individual accountability 
means “each student is assessed individually concerning his/her mastery of the concepts and the material involved in 
the group learning exercises” (Alavi, 1994, p. 165). Secondly, some researchers believe that cooperative learning 
and collaborative learning differ in their depth of interaction, integration, commitment, and complexity. When 
putting cooperative learning and collaborative learning on a continuum measured by levels of these four factors, 
cooperative learning falls at the low end of the continuum, whereas collaborative learning falls at the high end of it 
(Himmelman, 1996). Lastly, other than their implicational differences, studies on collaborative learning and 
cooperative learning tended to use different research methodologies. Studies on cooperative tradition tended to use 
quantitative methods that looked mostly on the learner achievement. Studies on collaborative learning, on the other 
hand, took mostly the qualitative approach by analyzing students’ interactions (Panitz, 1996, 1997). 

In general, both collaborative and cooperative learning involve the instructional use of small groups in 
which students work together to maximize their own and each other’s learning. Therefore, when a study’s focal 
point is rather the interactive group learning than the specific group learning approaches, it is legitimate to use these 
two terms without clarifying their differences. This study focused on the group interactive learning in general; 
therefore, it blurred the line between collaborative and cooperative learning and treats them as the same learning 
approach. 

Identity 

As Buckingham (2008) pointed out “identity is an ambiguous and slippery term” (p.l). There are various 
assumptions and definitions of identity based on disciplinary paradigms from which they are derived. In the 
sociology realm, the root for the concept of identity can be traced back to the early sociological interests in “self and 
society” (James E. Cote & Levine, 2002). Sociologists believe that identity is essentially a social and historical 
construct, and identities exist only when people participate in structured social relationships (Stryker, 1968). The 
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essential focus of collaborative learning is the social interactions within a learning group; therefore, the sociologists’ 
tradition of identity concepts and studies are appropriate for studies that explore collaborative learning from the 
perspective of identity. 

Structure of Identity 

Identity is multifaceted and dynamic (Brewer & Gardner, 1996). It should be viewed as an organized 
structure instead of a fixed unity (Stryker, 1968). The crux of social identity theory is the notion that individuals’ 
identities contain both personal and social components (Banaji & Prentice, 1994; Bettencourt, 1999; James E. Cote 
& Levine, 2002; Fearson, 1999; James, 1907, 1980; Mead, 1934; Turner, 1982). The social component of identity 
was later divided by researchers into two identity aspects: relational and collective identity (Brewer & Gardner, 

1996; Kashima & Hardie, 2000; Lord, Brown, & Freiberg, 1999; Sluss & Ashforth, 2007) to describe “those that 
derive from interpersonal relationships and interdependence with specific others and those that derive from 
membership in larger, more impersonal collectives or social categories” (Brewer & Gardner, 1996, p. 89). Relational 
identity generally refers to an individual’s sense of self in terms of his power, entitlement, and social affiliation in 
relation to others (Holland, Lachicotte, Skinner, & Cain, 1998). It is derived from the connections and role 
relationships with significant others such as relationships between children and their parents, students and their 
teachers, and employees and their bosses (Brewer & Gardner, 1996; Lord, et al., 1999). In a group setting, relational 
identity refers to an individual’s apprehension of his social position in the group derived from ties with other group 
members (E. S. Kashima & Hardie, 2000; Ozmantar, 2005). The idea of collective or group identity addresses the 
“we-ness” of a group and stresses the similarities and shared attributes of group members (Hardy, Lawrence, & 
Grant, 2005). Given together, an individual’s self-conception is composed of three aspects: individual, relational, 
and collective identity (Brewer & Gardner, 1996; E. S. Kashima & Hardie, 2000; Y. Kashima, et al., 1995; 

Sedikides & Brewer, 2001; Sluss & Ashforth, 2007; Triandis, 1989). 

Identity salience 

Identity salience theories believe that within an individual's multiple identities, only one identity can take 
the dominant position for a person in a given situation (Burke, 2003; Lord, et al., 1999; Stryker, 1968; Stryker & 
Burke, 2000). Identity salience defines the probability and readiness of acting out an identity across a variety of 
situations, or alternatively across persons in a given situation (Stryker & Burke, 2000; Stryker & Serpe, 1994). 

When combining the identity salience theory with the three identity aspect theory, it makes three different types of 
identity salience; individual, relational, and collective identity salience (Brewer & Gardner, 1996; E. S. Kashima & 
Hardie, 2000; Sluss & Ashforth, 2007). In other words, each individual has salient identity aspects among the three 
identity aspects. 

An individual’s identity salience is viewed as trans-situational and stable across time and situation (Stryker 
& Burke, 2000; Stryker & Serpe, 1994). A person will tend to perform according to his/her identity different a 
situations as well as seek opportunities to perform in terms of that identity (Stryker, 1968). 

Collaborative Learning and Identity Salience 

Although most studies have proved the positive effects of the collaborative learning approach, not all 
collaborative learning activities were successful ( Dillenbourg, et al., 1996). Many collaborative groups were 
ineffective and failed to generate any collaborative action, and some were even destructive (Hardy, et al., 2005; D. 
W. Johnson & Johnson, 1996). Because the increasing pervasion of the collaborative learning approach in today's 
education realm, it becomes more and more important to determine the demotivating factors of this learning 
approach in addition to its motivating factors. Researchers indicated that the failure of collaborative learning 
activities was mostly attributed to participants’ communication problems, perceived group statuses (Barron, 2003; 
Nuthall, 1999), effort-avoidance/social loafing (Salomon & Globerson, 1989), and the consequent group tensions 
(Smith, 2005). Researchers also pointed out that putting students together and telling them they were a group could 
not ensure the happening of collaborative learning (D. W. Johnson & Johnson, 1996; Slavin, 1999). The 
accomplishment of effective collaboration depends on the leverage of participants' differences, the balance of 
participants’ different concerns (Hardy, et al., 2005), and the achievement of intersubjectivity and common ground 
between participants (Ozmantar, 2005; Tudge, 1992). Intersubjectivity is the status of a shared understanding among 
group members (Bober & Dennen, 2001; Davidson, 1992; Dennen & Wieland, 2007; Fawcett & Garton, 2005; 
Summers, Beretvas, & Gorin, 2005). To this end, researchers claimed three distinctive and interrelated conditions for 
effective collaborative learning: group members’ mutuality, positive interdependence, and equality (Damon & 
Phelps, 1989; Dillenbourg, et al., 1996; Granott, 1993; Hardy, et al., 2005; Ozmantar, 2005; Slavin, 1996; Suzuki & 
Kato, 1997). Positive interdependence is achieved when participants feel responsible for one another’s learning and 
believe that their goals can only be attained when their team members also attain their goals (Alavi, 1994; Deutsch, 
1962; D. W. Johnson & Johnson, 1996; Soller, 2001). In other words, interdependence indicates the necessity of 
group members’ reliance on each other to accomplish their shared goals. The sense of interdependence leads to 
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promotive interaction in which participants encourage and facilitate each other’s efforts to complete tasks in order to 
reach the group's goals (D. W. Johnson, Johnson, & Stanne, 1989) through sharing of resources, information, and 
knowledge. Mutuality is concerned with a reciprocal bilateral communication amongst team members (Goos, et ah, 
2002). Equality means each participant takes other group members' perspectives and directions rather than a 
unilateral flow of direction (Damon & Phelps, 1989). 

Although researchers have pointed out the problems and conditions regarding collaborative learning, it is a 
comparative passive remedial approach. The fundamental causes of collaboration problems and motivational factors 
that contribute to effective collaborative learning have been left mostly untouched. 

Identities are motivators of human actions (Stryker, 1968). Different aspects of identity are associated with different 
social motivations and implications for an individual’s self-esteem and behavioral tendencies (Triandis, 1989). 
Therefore, students’ different identity salience may affect them to behave positively or negatively in collaborative 
learning activities. 

The individual identity salience orients towards one’s own interests and profit (Brewer & Gardner, 1996; E. 
S. Kashima & Hardie, 2000; Lord, et ah, 1999). It does not resonate well with the collaborative learning setting, 
which emphasizes collective efforts. As Stryker & Burke (2000) pointed out a mismatch or discrepancy of an 
individual’s identity salience results in negative emotion. Therefore, it may hinder the individual’s tendency to 
perform of that identity. Consequently, students with individual identity salience may not feel comfortable working 
in a collaborative setting. On the other hand, collective identity salience places a premium on the success of the 
group other than him own success or benefit (Brewer & Gardner, 1996; E. S. Kashima & Hardie, 2000; Lord, et ah, 
1999), while the relational identity salience values interdependence (Markus & Kitayama, 1991). Both of these two 
identity salience are more consistent with the collaborative learning setting. As a result, students with collective or 
relational identity salience may feel more comfortable working in collaborative groups. Based on this, the 
hypothesis of this study is that students with relational identity or collective identity salience have more positive 
attitude toward collaborative learning than students with individual identity salience. In addition, what factors will 
affect students taken of certain identity salience is also the purpose of the study. The main factors includes in this 
study are participants’ age and nationality. Some researchers claimed that different cultures tend to have nurture 
different identity salience. For example, Triandis (1989) and Singlis (1994) suggested that Asians tended to have 
collective and less individualist relative to Europeans or North Americans. Moreover, attitude is only influenced by 
an individual’s intentions to act with regard to the content, but also affected by the individual’s direct experience 
(Nabi & Krcmar, 2004). That is to say, the prior experience of collaborative learning also affects students’ attitudes 
toward collaborative learning. What specific experience matters is the focus of this study, such as being a leader, 
feeling of doing most the group work and so on. 

To this end. this study sought to answer one main quantitative research questions and two sub-questions: 1) 
Do students’ identity salience affect their attitudes toward collaborative learning? 2) Do students’ age and 
nationality affect students’ identity salience? 3) Do students’ prior experiences of collaborative learning affect 
students’ attitudes toward collaborative learning? 

Research Methodology 

Research Participants and Procedure 

Twenty graduate students in the school of education attended this study. The participants’ age ranged from 
20 to above 50 including: 10 (50%) in 20-30, five (25%) in 30-40; two (10%) in 20-50; and three (15%) above 50. 
Fourteen (70%) participants attended this study were first year graduate students. Eight (40%) participants were 
American, five (25%) participants were African American, four (20%) participants were international students from 
Asian, two ( 10%) participants were African, and only one (5%) participant was from South America. In order to 
draw a feasible but representative sample from target population, participants were randomly selected to include 
students from different age ranges and genders. 

Survey Instrumentation 

A survey questionnaire was used for this study. It consisted of three sections. The first section was adapted 
from the Kashima and Hardie (2000) RIC-scale survey to measure participants’ identity salience. The Kashima and 
Hardie RIC-scale has 10 items, and each of the items is followed by three response items corresponding to 
relational, individual, and collective identity aspect. The second section of the questionnaire included 10, 5-point 
Likert-scales ranging from strongly agree to strongly disagree to investigate participants’ attitudes toward peer 
collaborative learning and their prior experience in the collaborative learning activities. Questions in this section 
combine questions that developed by the researcher. The last section of the questionnaire asked participants to report 
their demographic and other school-related information. 
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Data Analysis Method 

One way ANOVA, simple regression, multiple regression, reliability analysis, and correlation were used to 
analyze the data. The main statistic test used for this study was ANOVA, a powerful and common statistical 
procedure in the social sciences (Plonsky, 2009). The regression statistic analysis was used to analyze the 
relationship between students’ prior experience in collaborative learning and their attitudes toward collaborative 
learning. 

Findings and Discussion 

According to the statistic description, among the 20 students that participated in the study, 1 1 (55%) 
students had individual identity salience, six (30 %) had relational identity salience, and only three ( 15%) had 
collective identity salience. The mean score of the attitude toward collaborative learning of the three identity groups 
were 0.5455 (individual), 1.3750 (relational), and 1.3333 (collective). That means all identity salience group had 
positive attitudes toward collaborative learning. 

In table 1, the ANOVA output was F=4.238 (p<0.05). It indicated that the difference of students’ attitudes 
toward the collaborative learning between three identity groups was statistically significant. As illustrated in table 2 
there was a statistically significant difference (p<0.05) between students with individual identity salience and 
relational identity salience toward collaborative learning. Student with collaborative identity salience consistently 
viewed collaborative learning more favorable than did students with individual identity salience. There was a 
statistically significant difference (p < 0.05). Collective identity salience was not significantly different to both 
relational identity salience and individual identity salience in terms of students’ attitudes toward collaborative 
learning. Since the mean score of collective identity (17.33) is similar to the mean score of relational identity (17.5), 
the insignificance may due to small number of the collective identity group (N=3) 

ANOVA 



attitude! 473Mean 





Sum of 
Sauares 


df 


Mean Sauare 


F 


Sid. 


Between Groups 


3.297 


2 


1.648 


4.238 


.032 


Within Groups 


6.61 3 


17 


.389 






Total 


9.909 


19 









TABLE 1: ANOVA output 
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.056 
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collective 


.04167 


.44101 


.996 


-1.1403 


1.2236 
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individual 


.78788 


.40623 


.183 


-.3009 


1.8766 






relational 


-.04167 


.44101 


.996 


-1.2236 


1.1403 



* The mean difference is significant at the 0.05 level. 



TABLE 2: multiple comparison among three identity groups 

As indicated in table 3 and table 4 students’ identity salience were not significantly related to their age 



ranges ( % =5.545, p= 0.476 > 0.05) or the countries ( A =15.386, p= 0.052 > 0.05) where they were from. 

2 

However, the (0.05. df =8) =15.51. It was very close to 15.368. Because this study only had 20 participants, 
adding new participants might change the result. 
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Crosstab 



Chi Square Tests 
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50- Total 


identity 


individual 


7 


2 1 


1 11 




relational 
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2 0 


1 6 




collective 


0 


1 1 


1 3 


Total 




10 


5 2 


3 20 



TABLE 3: output for age and identity relationship 



Asymp. Sig. 
Value df (2-sided) 



Pearson Chi-Square 


5.545' 


6 


.476 


Likelihood Ratio 


6.852 


6 


.335 


Linear-by- Linear 
Association 


2.841 


1 


.092 


N of Valid Cases 


20 







a. 1 1 cells (91 .7%) have expected count less than 5. The minimum expected count 
is .30. 
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3 
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2 


0 
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5 
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Chi-Square Tests 




Value 


df 
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(2-sided) 


Pearson Chi-Square 
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8 


.052 


Likelihood Ratio 


18.953 


8 


.015 


Linear-by-Linear 

Association 


1.666 


1 


.197 


N of Valid Cases 


20 







a. 1 5 cells (1 00.0%) have expected count less than 5. The minimum expected 
count is .15. 



TABLE 4: output for nationality and identity relationship 



As indicated in table 5, in general, the relationship between students’ prior experience in collaborative 
learning and their attitudes toward the collaborative learning approach was statistically significant (F=5.868, p= 
0.004 <0.05). Item2 (“doing most of the work”; t=-3.029, p=0.009 < 0.05) and Item 6 (“explaining of ideas to group 
members”; t=2.789, p=0.014<0.05) were significantly related to students’ attitudes toward the collaborative learning 
approach. 

Coefficients' 
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2.539 
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doing most of the work 
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.009 


good relationship with 
peer 
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explain ideas 


2.719 


.975 


.579 


2.789 


.014 


frequent communication 


-.206 


.884 
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-.233 


.819 


role awareness 


-.185 


.889 


-.073 


-.208 


.838 



a. Dependent Variable: atitudeTotall 478 



TABLE 5: Coefficient table of students’ prior experience in collaborative learning 



As shown in table 6, 67.7% of students’ attitudes toward collaborative learning can be predicted by 
students’ prior experience of collaborative learning. 

Model Summary 1 ’ 













Chanqe Statistics 




Mode 
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R R Square 


Adjusted R 
Square 


Std. Error of 
the Estimate 


R Square 
Chanqe 


F Chanqe dfl df2 


Siq. F Chanqe 


1 


.823' .677 


.562 


1.91265 


.677 


5.868 5 14 


.004 



a. Predictors: (Constant), role awareness, doing most of the work, good relationship with peer, explain ideas, frequent communication 

b. Dependent Variable: atitudeTotall 478 
TABLE 6: Model Summary table 
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Conclusions 

This study explored the relationship between students’ identity salience, prior collaborative learning 
experiences, and students’ attitudes toward collaborative learning. The research findings were consistent with the 
study hypothesis that students’ attitudes toward collaborative learning were affected both by students' identity 
salience and their prior collaborative learning experience. Three different identity salience groups all had positive 
attitudes toward collaborative learning; however, the relational identity salience group had significantly more 
positive attitudes toward collaborative learning than the individual identity salience group. In addition, students’ 
experience of doing most of the group jobs was negatively related to students’ attitude toward collaborative learning, 
while experience of explaining ideas to group members was positively related to students’ attitudes toward 
collaborative learning. At last, the findings also indicated that students’ identity salience were not related to their age 
ranges and nationalities. 

This study provides researchers with a new perspective to explore effectiveness in collaborative learning 
settings. It also provides a new factor, identity salience, for instructors to consider when grouping students for 
collaborative learning activities, and cultivating student toward team work. There are two major limitations of this 
study. First, the sample size of the study is small. It limits the generalization of the findings. For the future study, a 
bigger sample size is desired. Second, there were only six questions asking about students’ prior collaborative 
learning experience. The limited number of the questions may have excluded other possible experiences that affect 
students’ attitudes toward collaborative learning. 
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Abstract 

Technology-based learning environments offer new ways to support learning in complex domains, but there is still a 
lack of understanding as to under what conditions learning is most likely to be successful. Moreover, reliable and 
valid methodologies for assessing the learners’ performance are required. We designed and developed a simulation- 
based learning environment in the area of school management which includes (1) instructional support to facilitate 
learning, (2) collaborative features, and (3) a methodology to determine the learning outcome. A study with 66 
participants was conducted to evaluate effects of the learning environment on learning outcome and validation of the 
used instruments. Results showed that students’ performance significantly improved after using the learning 
environment and they had no problems with the complex domain. Data also revealed validation issues for the 
applied instruments. We discuss our initial findings and conclude with directions for future research. 

Introduction 

The rise of computer technology affected the design of learning environments and raised hopes and 
expectations regarding effective instruction. Features such as the multimedia capability and networking 
opportunities of computers opened new paths for applying instructional principles in many ways. However, 
technology alone will not lead to successful learning and it is more the underlying pedagogy than the technology 
that will facilitate learning which requires thorough design of effective learning environments. And even though 
technology-based learning environments offer opportunities to support learning in complex domains, there is still a 
lack of understanding as to under what conditions learning is most likely to be successful (Cannon-Bowers & 
Bowers, 2008). Moreover, to evaluate effectiveness of a learning environment, methodologies for assessing the 
learners’ performance are required. 

One possible application of technology to enhance learning involves the use of simulations, which play an 
important role in education. Clariana and Strobel (2008) define a simulation as “an executable (runnable) model, 
computer software that allows a learner to manipulate variables and processes and observe results” (p. 330). 
Simulations are particularly useful for learning about complex domains, because they enable learners to form and 
test hypotheses in a setting as complex as the field of application, but without risking negative consequences when 
making mistakes. Learners do not per se have the ability to construct meaningful and desired outcomes when 
working with complex problems (Dorner, 1996; Spector, 2000). Thus, we claim that learning environments which 
utilize technological enhancements such as simulations are only effective if instructional support is provided. 

Moreover, determining the effectiveness of learning environments requires well-defined performance 
criteria and valid measurement instillments. Christensen, Spector, Sioutine, and Cormack (2000) claim that learning 
should be assessed through a comparison of learners’ thinking and performance with that of experts. 

We developed a simulation-based learning environment in a complex domain which includes (1) 
instructional support to facilitate learning, (2) collaborative features, and (3) a methodology to determine learning 
outcome. Our goals were firstly to examine the effect of a simulation-based learning environment on students’ 
learning outcome and secondly to develop a methodology for assessing students’ performance when working on 
complex problems. 

The next section will focus on rationales for the use of technology-based learning environments including 
our research questions and hypotheses. Then we will introduce the design and procedure of our research study, 
followed by the presentation of our results. Finally, we will discuss our findings and present directions for future 
research. 

Rationales for the Use of Technology-based Learning Environments 

The mere use of technology alone does not lead to successful learning, but computer-based learning 
environments have the potential to facilitate authentic learning where students can construct meaningful knowledge. 
When providing learners with authentic activities, knowledge is not presented in abstract forms, but rather 
embedded in a meaningful context. Learner-centered activities play a central role where the teacher’s role moves 
from a directive disposition to a supportive one. This means, that instruction is not broken down into manageable 
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parts and then presented to the learner, but rather experienced by the learner under the teacher’s guidance (Harley, 
1996). Learning is an active process where students simulate and recreate real-life complexities without a simplified 
knowledge base (Karagiorgi & Symeou, 2005). Students do not absorb decontextualized knowledge and skills, but 
learning is situated which means that it occurs in context, because learners build personal interpretations of the 
world with knowledge emerging in the context within which it is relevant (Lave & Wenger, 1991; McLellan, 1996). 
Advocates of situated learning approaches argue that students are not prepared effectively for job performance if 
learning is separated from practice (Enkenberg, 2001) and claim that decontextualized knowledge also contributes to 
the problem of inert knowledge which refers to information that a learner may possess, but is not able to recall or 
apply in order to solve problems (Whitehead, 1929). 

Computer-based learning environments can be effective tools when it comes to implementing situated 
learning activities. One possible application is the use of simulations of complex systems as a means for training and 
research. Educational technologists have been employing computer-based simulations and games in a variety of 
educational contexts to utilize technology in supporting effective instruction. Simulations have become a prevalent 
tool in learning and instruction since they provide a broad applicability in different instructional settings (Ifenthaler, 
2009). The following elements are essential to an effective instructional simulation: complexity, learning objectives, 
interaction, real-world processes, and embedded, and non-random outcomes (Shelton, 2007). 

Additionally, the meaning making process in situated learning environments often includes collaboration 
between learners, because learning is a social, dialogical process based on the negotiation between learners 
(Jonassen, Davidson, Collins, Campbell, & Haag, 1995). The influence of social interaction and individual meaning 
making is reciprocal, because personal understanding and social knowledge building are interdependent and cannot 
exist separately (Stahl, 2006). A different kind of interaction can be found in a continuing acquisition of knowledge 
and skills when novices learn from experts in the context of practical activities. Experts can either act as role models 
or coaches or both. Lave and Wenger (1991) propose to let the participant move from being an observer to a fully 
functioning agent within the culture of the group. This will enable learners to learn how to speak about and within a 
community of practice. 

Instructional designers need to be developing learning environments which enable learners to construct 
meaningful knowledge. This can be achieved by providing learners with collaborative activities that are embedded 
in a context meaningful to them with room for reflection with others about their learning progress (Jonassen et al., 
1995). An example for an instructional model that applies principles derived from the situated learning approach 
such as learning activities and collaboration is problem-based learning (Jonassen, 1997; Savery & Duffy, 1995a, 
1995b). Problem-based learning describes an instructional method where learning occurs when learners are working 
on a solution for a given, authentic problem. Learning is considered the process of creating content knowledge and 
developing problem-solving as well as self-directing skills (Hung, Jonassen, & Liu, 2008). Other models 
implementing principles from the situated learning approach are cognitive apprenticeship (Collins, Brown, & 
Newman, 1989), project-based science (PBS), and communities of practice (Lave, 1991). Cognitive apprenticeship 
refers to a way of learning, where novice learners first observe an expert and are then guided by a more experienced 
person to utilize their own cognitive and metacognitive skills. The experienced person may demonstrate the skills 
and levels, provide examples, and support the novice through feedback and assistance (Dennen & Burner, 2008). 
Project-based science is characterized by a driving question, investigations, artifact development, collaborations and 
use of tools to support inquiry (Singer, Marx, Krajcik, & Chambers, 2000). A community of practice (CoP) is a 
sustained social network of people which can be bound in a formal or informal way. They are engaged in a common 
practice and share a common set of values and knowledge (Dennen & Burner, 2008). All approaches have in 
common that learners actively work on a complex task in interaction with peers and/or a more experienced person. 
The teacher is in the role of a coach who provides information and support when needed. 

The Learning Environment 

Our modular learning environment is concerned with the area of school management. It was derived from 
principles of the situated learning approach and includes questionnaires to assess dependent variables, instructional 
materials, communication tools, and a simulation which is embedded in an authentic scenario. Students work 
individually as well as collaboratively. 

The learner’s mission is to improve a school’s efficiency, which is represented by means of the school’s 
dropout rate. Student retention is an important issue in schools today (Attewell & Seel, 2003). Many factors which 
lead a student to drop out of school may be found in the students themselves, such as socio-economic status, their 
parents’ level of education, etc. (Grob, Jaschinski, & Winkler, 2003). However, some studies have revealed that 
changing the school itself rather than the student may contribute to preventing at-risk students from dropping out of 
school. Schools are systems with many interrelated variables which contribute in different ways to dropout rates. 
Leithwood, Lawton, and Cousins (1989) identified seven categories that are indirectly and directly related to 
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dropout, such as teachers, programs & instruction, and school-community relations. Their empirical model was used 
as the underlying causal model of the simulation (see Ifenthaler, 2009). Complex systems are characterized by 
interconnected variables which influence each other (Funke, 1991; Seel, Ifenthaler, & Pirnay-Dummer, 2009). In the 
simulation, all seven categories of the underlying model are represented as investment fields, i.e. learners can 
allocate money from a given budget to different areas with each standing for a particular attribute of one variable of 
the causal model. 

The main learning objective is to explore the underlying causal model of the simulation. Students can enter 
data into the simulation and receive an output, but calculations as well as causal relationships between the variables 
are not known by the user. Thus, students need to infer the causal model from the output. To facilitate the 
exploration, they will be given information about the variables which allows for conclusions regarding their 
interrelationships. Moreover, students will learn about the underlying model through feedback from the system 
(simulation output such as the school’s dropout rate) and collaboration with peers. The collaborative part enables 
them to exchange ideas, reflect on past results, and further refine their mental model. Collaboration may facilitate 
learning in complex domains, but evidence about its effectiveness has not yet been demonstrated clearly. 
Nevertheless collaboration is effective insofar as it enhances active learner engagement, which is generally crucial to 
improved learning outcomes (see Christensen et ah, 2000). 

Research Questions and Hypotheses 

According to Johnson-Laird (1989), it is important to understand the learning-dependent progression from 
a novice’s to an expert’s understanding of a complex domain (Ifenthaler & Seel, 2005). Therefore, we will compare 
the learner’s understanding of the underlying model of the simulation with that of the expert to assess the final 
learning outcome. Accordingly, we will address the following research questions in our empirical investigation: 

1 . Is the simulation-based learning environment an effective instructional method for improving the 
understanding of complex domains (systems)? 

2.1s the comparison of novice models and expert models a valid instrument for assessing learning- 
dependent progression in complex domains? 

We claim that our simulation-based learning environment is an appropriate tool for enhancing students’ 
complex problem solving skills. Interacting with a simulation that models the behavior of a complex dynamic 
system enables learners to create and test hypotheses in order to refine their mental model about the problem. 
Collaborating with peers provides learners with new perspectives, facilitates reflection, and thus helps them to 
develop a solution to the given problem. We hypothesize that students will perform better on the posttests than on 
the pretests. 

We argue that learning can be viewed as moving along a continuum towards expertise. Therefore, the 
similarity between students’ models and the expert model (the causal model of the simulation) should increase, thus 
indicating improved learning. To measure these outcomes, we have two different instruments. The simulation 
output, which calculates the new dropout rate of the school, provides a direct indication about the quality of a 
learner’s decision. Success is reflected by decreased dropout rates and failure by increased dropout rates. Secondly, 
a questionnaire will be given to the learners in which they will be asked to rate the impact of the variables on each 
other and on dropout rates. The data from the questionnaire will be transformed into a causal model which will then 
be compared to the expert model, i.e. the underlying model of the simulation. The higher the similarity score, which 
ranges from 0-1, the more similar are the models. 

Method 

Participants 

The sample consisted of 66 students (54 female and 12 male) from a German Gymnasium (high school). 
Their average age was 17.94 years ( SD = 1.28). It included three classes from grades 11-13. The Gymnasium 
participating in the study is specialized in pedagogy, which means that the area of school management is part of the 
students’ curriculum, might be related to future job opportunities, and is likely of personal interest to them. 

Design and Procedure 

The study was conducted during two regular 90 minute class sessions and was divided in four main phases 
(see Table 1). The study started with a short introduction in phase 1 and then students logged into the online learning 
environment and completed a demographic survey and pretests. Then they were given more information about the 
study and scenario along with instructions in class. The students’ first learning task was to work through the 
provided instructional materials and learn about the categories involved in school management in phase 3. Students 
were randomly assigned to discussion groups of five for the collaborative part in phase 3. During the second 90 
minute class session, students were given 60 minutes to discuss their results and collaboratively explore the causal 
model of the simulation in order to achieve an optimal result in the last simulation run. After the collaborative 
activities, the study closed with the posttests and an evaluation survey (see Table 1). 
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In the evaluation survey, students were asked to evaluate different aspects of the learning environment on a five- 
point Likert scale (1=1 disagree, 5=1 fully agree). It served to assess usability of the learning environment, the 
benefit of instructional support, and success of the collaboration. Both pretest and posttest consisted of two scores 
that will be further explained in the following paragraph on outcome variables. 



Table 1 Study phases and components 



Phase 


Activities 


Obtained Data 


Point in time 


Duration 


1 


Pretest and demographic 
survey 


dropout rate 

causal model similarity 
score 


Day 1 


45 min 


2 


Learning activities I: 
individual tasks 




Day 1 


45 min 


3 


Learning activities II: 
collaborative task 




Day 2 


60 min 


4 


Posttest and evaluation 
survey 


dropout rate 

causal model similarity 
score 

students’ evaluations of the 
learning environment 


Day 2 


30 min 



Outcome Variables 

Two outcome variables were used for this study, namely (1) dropout rate and (2) causal model similarity’ 
score. The simulation returned a new calculated dropout rate after each run based on the learner’s input which 
provides performance information. Good decisions from the learners and effective placement of investments will 
lead to a lower dropout rate. Non-effective investments result in an increased dropout rate. This means that the lower 
the dropout rate, the better the performance. Participants were asked to rate the impact of variable that are part of the 
causal model underlying the simulation on each other with a score between 1 and 10. Each score was transformed 
into a directed relation. All obtained relations combined resulted in a causal network. This was then compared to the 
causal model underlying the simulation using the SMD (Surface, Matching, Deep Structure) software developed by 
Ifenthaler (2008b). The software calculates a score ranging from 0 (no similarity) to 1 (identical) indicating 
similarity of a participant’s causal model with the underlying model of the simulation. The closer a participant’s 
causal model is to the expert model, i.e. the underlying model of the simulation, the better the person performs. 

Results 

The collected data were quantitatively analyzed to first explore the effectiveness of the simulation-based 
learning environment on students’ learning outcome. It was expected that students performed better on the posttests 
than on the pretests. Second, a correlation analysis between the two outcome variables, dropout rate and similarity 
score, was conducted to cross-validate the two measurement instruments. 

Effectiveness of simulation-based learning environment 

A prior version of the simulation (DIVOSA; see Ifenthaler, 2002, 2009) had been used before and showed 
promising results. The instructional materials along with the collaborative part should have enhanced students’ 
understanding of the complex problem, namely school management. We thus expected indicators for learning from 
pretest to posttest reflected by a higher similarity between the learner’s model and the expert model on the one hand 
and a lower dropout rate on the other hand. Paired t-tests were conducted to assess learning gains (posttest - pretest 
comparison). The descriptive data included in Table 2 show that the mean 6.86 ( SD = 2.23) of the dropout score in 
the pretest is higher than the mean 5.71 (SD = 1.81) of the score in the posttest. 

Table 2 Paired samples statistics 

Dropout rate N M SD 

pretest score 66 6.86 2.23 

posttest score 66 5.71 1.81 

Note. A lower score indicates better performance 
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A paired-samples t-test was employed to test the hypothesis that there are differences between the dropout 
rate obtained in the posttest compared to that in the pretest. The t-test revealed significant differences between the 
dropout rates obtained in the pretest and those from the posttest with M difference =1.15 (SD = 2.54), 1(65) = 3.69, 
p < .01, d= .61 (medium effect). Therefore, the data support the stated hypothesis which means that students 
performed better on the posttest than the pretest. 

Not all students completed the questionnaire for the similarity score and therefore, six datasets had to be 
excluded from the analysis due to the missing values. The descriptive data of the remaining sample (/V rcm = 60) in 
Table 3 show that the mean .248 (SD = .179) of the posttest causal model similarity score is higher than the mean 
.230 (SD = .178) of the pretest causal model similarity score. 



Table 3 


Paired samples statistics 




Similarity score 


N 


M 


SD 


pretest score 


60 


.23 


.18 


posttest score 


60 


.25 


.18 



A paired-samples t-test was employed to test the hypothesis that there are differences between the 
similarity score obtained in the posttest compared to that in the pretest. The t-test revealed no significant differences 
between the dropout rates obtained in the pretest and those from the posttest with M difference = .02 (SD = .22), 
t(59) = .59, p = .56. This means that the data do not support the hypothesis and the similarity score did not improve 
during the study. 

Table 4 shows the results of the survey where students were asked to evaluate the learning environment 
with 1 representing a low score and 5 a high score. The survey covered three different areas, namely usability, 
instructional support, and collaboration. Each of them was represented by several items as presented in Table 4. The 
data show that students positively rated the learning environment since all three factors received high average 
ratings with acceptable standard deviations. 



Table 4 Statistics of the evaluation survey 



Factor 


No. of 
Items 


M 


SD 


usability 


21 


3.80 


.48 


instaictional support 


14 


3.63 


.45 


collaboration 


13 


3.40 


.56 



Comparison of expert and novice models for assessing learning gains 

It was argued that learning is regarded as a progression from novice to expert models. We assumed that a 
success in the simulation game (low dropout rate) will be associated with a high similarity score between the 
learner’s model and the expert model (simulation’s causal model). Thus, we expected a high validity of our research 
instalments using a cross-validation approach between the two instalments (simulation output, and causal model 
assessment) respectively. We expected a negative correlation between the two measurement instalments. 
Correlations between the two scores were calculated and revealed a positive correlation of r = .26 between the two 
posttest scores and a negative correlation r = -.03 between the pretest scores. None of the two correlations were 
significant which indicates that the there is no correlation between the measurement instruments. The results raise 
concerns regarding validity of the two applied measurement instruments that requires further examination which 
will be taken on and discussed in the discussion section. 

Discussion 

Purpose of this study was to examine effects of a simulation-based collaborative learning environment as 
well as methodologies to assess learning outcome. The learning environment included instructional materials, 
communication tools, and a simulation which was embedded in a complex, authentic scenario. The content evolved 
around the topic area of school management and learners were asked to reduce the dropout rate of a school and thus 
to improve its efficiency. 66 students from three classes from a German gymnasium specializing in pedagogy 
participated in the study. Two outcome variables were used to assess students’ performance. The first one was the 
dropout rate returned by the simulation with a low score indicating success. The second one was a score representing 
similarity between students’ causal models and the simulation’s underlying model with a high similarity indicating 
success. 
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The first research question concerned the effectiveness of the learning environment and was addressed by a 
pretest-posttest comparison for each of the two outcome variables. Additionally, an evaluation form was given to 
students to rate usability of the learning environment. Results varied between the first outcome variable, the dropout 
rate and the second outcome variable, the similarity score. On the one hand, paired samples t-tests on the first output 
variable showed that students were able to significantly reduce the dropout rate of the simulation from pre- to 
posttest thus indicating learning gain. On the other hand, paired samples t-test on the second output variable, showed 
that the similarity between students’ causal models and the simulation’s underlying model did not increase. This 
means that students’ performance did not improve after students began working with the simulation-based learning 
environment given the assumption that that learning is a process whereby novices move towards expertise. 

At first, it seems odd that the two outcome variables revealed contrary results regarding the effectiveness of 
the learning environment. The contrary results also affect the second research question concerning the applicability 
of the used instmments for assessing learning-dependent progression in complex domains. Results revealed that a 
cross-validation between the two instruments failed since no correlation between the scores was found. A possible 
explanation might be validity problems with either one of the outcome variables or both. The following five reasons 
suggest that the similarity score has validity issues: first, software errors occurred during that data collection process 
which resulted in missing datasets. Second, students asked many questions when filling out the online questionnaire 
which indicates that the interface needs improvement in its user friendliness and usability. Third, students reported 
that the similarity questionnaire made not much sense to them which might be a result of missing prerequisites to 
work with that instrument. They were not particularly trained regarding system thinking, modeling, and other 
prerequisites that might be necessary to understand a school and the involved areas such as teachers and curriculum 
as a complex system that consists of interrelated components. Fourth, the diagnosis of internal knowledge structures 
is one of the most challenging tasks and remains unsolved in many respects (Al-Diban, 2008; lfenthaler, 2008a). 
Measuring an internal encoding requires the process of externalization which can be seen as a re-representation of 
the person’s understanding about the world. Each assessment technique may exacerbate the externalization process 
and thus lead to invalid results (lfenthaler, 2008a, in press; lfenthaler, Masduki, & Seel, 2009). A fifths reason for 
the validity problems with the similarity score might be, that students’ mental models might be organized in a way 
that enables them to operate and understand the simulation, but differs from the simulation’s underlying model. This 
would explain why students were able to reduce the dropout rate without increasing the similarity of their causal 
network to the model underlying the simulation. There might be a knowledge structure or mental model that enables 
students to successfully manage the school not being analogous to a causal network like the one underlying the 
simulation. 

Based on the assumption that the similarity score is not a valid measure, but the dropout rate is a valid 
measure for students’ performance in the present study, we claim that our simulation-based learning environment is 
an effective learning tool. The significant reduction of the dropout rate from pretest to posttest suggests learning 
gains. Students’ positive attitude towards the learning environment reflected by the evaluation questionnaire 
indicates an adequate challenge level, which means that students did benefit from this particular implementation of 
technology in education. Students were coping well in the new territory and seemed to enjoy the complex, authentic 
activities. These findings support the benefit auf authentic activities and are in alignment with researchers who 
advocate inquiry-based learning and recommend to have students work on complex problem-solving rather than 
providing them with well-defined tasks (Collins et al., 1989; Harley, 1996; Jonassen, 1997; Karagiorgi & Symeou, 
2005; Savery & Duffy, 1995b). 

Future research will include a further development of our learning environment. Based on our empirical 
findings, we will continue developing adequate assessment techniques and investigate their effectiveness in future 
research studies. 
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Introduction 

In order to increase student engagement outside the classroom as well as providing a repository of 
resources for independent learning, Bilkent University School of English Language (BUSEL) preparatory 
program in Ankara, Turkey established a project that would integrate the use of computers into its existing 
foreign language curriculum. The project started by identifying its own goals based on the institutional mission 
statement of the preparatory program, which are to: 

• enable students to make the successful transition from life at school and home to life at 
university; 

• ensure that students attain the level of proficiency in English necessary to enter their 
chosen School or Faculty; 

• develop students’ potential as independent ‘autonomous’ learners; 

• support the further development of students’ English language and study skills throughout their 
study in the faculties and vocational training schools. 

The study was conducted in the English Preparatory Program in BUSEL, whose aim is to fully equip 
students with the necessary language skills that are required in their departments. In the system, there are five 
levels: Elementary, pre-intennediate, intermediate, upper-intennediate and the advanced level. For each level 
there is eight weeks of instruction, each of which is called a ‘course’. During one course the students are given 
two progress tests which determine whether they can sit the achievement test to pass to a higher level. If the 
students have an overall sixty percent they can sit the achievement test. At the end of the advanced level, the 
students take the proficiency exam and if they get sixty percent from the proficiency test, they start their 
departmental studies. 

The computerized learning project was established in the first academic semester of 2002-2003 by the 
researcher who designed, developed, and implemented the computerized learning architecture together with 
materials writers, who provided content for the level strands. The computerized architecture consisted of two 
tracks. Track 1, a discrete skills strand that was initially developed for two levels, both intermediate and upper- 
intennediate in the English preparatory program, consisted of discrete skills, i.e. listening, reading, and language 
components, i.e. grammar and vocabulary. This track accompanied the in-house coursebook in a unit-by-unit 
fashion and it was based on the prioritized language objectives for both of the levels. Track 1 focused on reading, 
listening, vocabulary and grammar and was designed in tandem with the institutional course book. Track 2 was 
an extended project strand that made use of in-house produced WebQuests, which were based on a common 
theme as the coursebook and Track 1 units and exposed students to language in an integrated manner. Students 
read and listened to texts, watched videos or movies about the topic and worked towards outcomes such as 
presentations, class debates and written work, where they were expected to synthesize the input materials they 
had received. 

Background to the Study 

The purpose of this study was to depict the existing computerized learning architecture of the program 
as utilized by the groups in the program, i.e. the students, teachers and administrators, and identify its perceived 
benefits on learning and teaching English as a foreign language as well as on transferable skills, i.e. computer 
literacy, higher-order thinking skills, handling of infonnation and working collaboratively. The study also aimed 
to identify areas that would need further improvement as a result of the evaluation conducted. 
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The study was inspired by two of Kirkpatrick’s (1994) 4 levels of evaluation. It was mainly influenced 
by Level 1 Reaction and Level 4 Results. However, the levels were not followed exactly in a sequence moving 
from one to another as suggested by Kirkpatrick himself. The influence from the two levels was mostly on 
detennining the areas to be evaluated: In this particular study, one major area for evaluation was satisfaction of 
all those involved in the implementation of the computerized learning architecture, which corresponds with 
Level 1. The other major area for evaluation was the effects of Track 2 as perceived by faculty students, which 
corresponds with Level 4, results. However, the results level, which has been designed particularly for business 
contexts to measure increased production, higher profits or decreased costs, was adapted for this study’s context 
in that it was used to explore the evaluation of results gained through Track 2, i.e. transfer of the skills, through 
the perception of students. Also one of the original underlying principles of this level, measuring the cost- 
benefit balance, was adapted in this study as an attempt to find out about the worth of the investment of time and 
resources in the learning architecture through the perceptions of end users, which would then lead to further 
design and development decisions of the architecture. Level 2 Learning has been deliberately left out as it would 
be very difficult to measure the level of learning that could be attributed to the use of the computerized 
architecture under the given circumstances. The use of this level has been recommended for further research in 
more controlled environments. Also it was believed that evaluating the learning would be quite a tedious effort, 
which could mean going back to the media or method dilemma (Clark, 1994; Kozma, 1994) so this aspect of the 
evaluation was left out to avoid media comparison. It was also decided that Level 3 Behavior also needed to be 
assessed under controlled conditions shortly after the implementation of the computerized learning program, 
which was not possible during the course of the study, so it was left out as well. 

The problem statement of this study was: 

How effective are the computerized learning architectures used at two levels of English language 
preparatory program of a private university, which are called Track 1 and 2, in terms of their 
contribution to the students’ language development and the enhancement of their transferable 
skills according to the perceptions of those involved in its implementation? 

This study investigated the following research questions: 

• What is the general reaction in the English preparatory school towards the use of computers through 
Tracks 1 and 2 in learning and teaching English as a foreign language? 

o What are the students’ perceptions of the potential benefits of Tracks 1 and 2 for 
their language skills? 

o What are the students’ perceptions of the potential benefits of Track 2 for the 
development of their transferable skills? 

o What are the instructors’ perceptions of the potential benefits of Tracks 1 and 2 for 
students’ language skills? 

o What are the instructors’ perceptions of the potential benefits of Track 2 for the 
development of students’ transferable skills? 

o What are the administrators’ perceptions of the potential benefits of Tracks 1 and 2 
for students’ language skills? 

o What are the administrators’ perceptions of the potential benefits of Track 2 for the 
development of students’ transferable skills? 

• What skills do faculty students perceive they can transfer to their studies in their departments as a result 
of being involved in computerized language projects through Track 2? 

o What are faculty students’ perceptions of the transferability of the skills they gain through 
Track 2 to their studies in their departments? 

The evaluation conducted through this study was significant for three reasons. It would: 

• help the computerized learning coordinator, materials writers, and instructors have a better 
understanding of the use of Computer Aided Language Learning (C ALL), which would in turn enable 
them to further develop and integrate it to the remaining levels in the preparatory program; 

• help the computerized learning coordinator, materials writers, and instructors find out how the CALL 
architecture was being used in reality and compare their own designs against their actual use. This 
could help identify the differences in perception and then help build a common vision of CALL and 
lead to effective implementation. It would also provide the kind of information that the senior 
management would need for decision making about the future of CALL in the institution; 

• contribute to the literature in the field by providing an insight from a specific case of implementation. 

Literature on Evaluation of CALL 

Evaluation in foreign language teaching initially seems to have followed an outcome-oriented approach 
and focused on effect studies (Chapelle, 2001). Most of the studies seem to have focused on the use of a single 
aspect of language teaching, that is either one skill such as reading or writing; another trend has been to focus on 
students’ perceptions of the programs’ effectiveness, which would be equivalent to Kirkpatrick’s (1994) Level 1 
evaluation: Reaction. Levy and Stockwell (2006) categorize the evaluation carried out on CALL materials in 
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three major categories in terms of their focus: a designer-evaluator focus, a language-skills focus, and a student- 
courseware focus. They also categorize CALL evaluation for large-scale frameworks as methodology driven, 
e.g. Hubbard’s use of “an approach checklist, a learner strategy checklist and other considerations” (as cited in 
Levy & Stockwell, 2006, p.58), or theory-driven, e.g. Chapelle’s use of the second language acquisition (SLA) 
theory as a basis of research (2001). This study had a particular designer-evaluator focus, and a language as well 
as transferable skills focus. 

Despite the interest in evaluation of CALL through numerous local and international studies, the 
effectiveness of the use of computers in improving second language competence cannot be generalized as it is 
now a fact that C ALL covers a variety of activity types that may involve or require other skills, not just simply 
clicking on words or filling the blanks; moreover, second language competence consists of complex and 
interrelated competencies, which means it might not always be possible to ascribe the favorable results to 
learning through the use of computers; third, the importance of the processes in language learning, not just the 
product, is now recognized as a justifiable reason for research; lastly, student characteristics and preferences 
have a significant effect on CALL and how it is utilized (Chapelle & Jamieson, 1989). Therefore, the aim of this 
particular study was to focus on the specific institutional use of Track 1 and 2 and their perceived results on 
language and transferable skills from the view points of all those involved in its implementation. 

Method 

Participants 

For research question 1, the accessible number of participants for the questionnaires was 896 in total, of 
which 497 were intermediate students, and 399 were upper -intermediate students. Also for interviews 22 students from 
this group were selected through maximum variation sampling. For research question 2, the participants included 14 
students for the interviews who had completed either intermediate or upper-intermediate Track 2, passed the 
proficiency exam and had started studying in their departments four months before they were contacted. This group 
was selected through criterion based sampling in that only students who took part in at least one Track 2 project were 
selected. 

As for the instructors in this study, all of the class instructors of the target classes, which was 69 in total, 40 in 
intermediate and 29 in upper-intermediate, were asked to fill in the instructor questionnaires. The administrators, all of 
the heads of the teaching units responsible for the two levels, 3 intermediate and 3 upper-intermediate, were given the 
head of teaching questionnaire. 

Table 1 Number of participants in the study according to research questions 





Research Question 1 


Research Question 2 


Students in 
the program 


Questionnaires 


Interviews 




896 


22 


Instructors 


69 


Administrators 


6 


Students in 
faculties 




14 



Methodology 

This was a case study that focused on evaluation through the use of mixed methods of collecting data. This 
evaluation was based on many of the characteristics of evaluation studies listed by Levy and Stockwell (2006). 
It: 

• was aimed at establishing the worth of the computerized learning architecture of the 
preparatory school 

• was primarily decision-driven 

• drew value from the process as well as from the product of the evaluation 

• focused on “Did it work?” (p.42) 

Johnson and Christensen (2004) categorize case studies into three groups: the intrinsic case study, in which 
the interest is in understanding a specific case, the instrumental case study, which focuses on more than the 
specific case being studied and the collective case study, in which several cases are examined in one study. This 
particular study falls under the category of intrinsic case study, which focused on the computerized learning 
architecture of a preparatory English program with the purpose of having an overview of the perceptions of the 
participants so as to make informed decisions for both short and long term planning. 

The evaluation in this study was summative evaluation which was applied to the whole computerized 
learning program by using survey instruments, which are common tools for CALL evaluation (Levy & 
Stockwell, 2006, p.43) and observations. 
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Johnson and Christensen (2004) explain that case study methodologists advocate using several methods 
and several data sources and assuming an eclectic approach for research. Similarly, the data for this study has 
been collected through using mixed methods, which is a questionnaire with a numerical part, as well as an open- 
ended part, two sets of interviews and laboratory and classroom observations. Mixed methods have been chosen 
for the way they can explain the relationship between variables, allow the researcher to study the relationships in 
depth with qualitative data and “confirm or cross-validate relationships discovered between variables” (Fraenkel 
& Wallen, 2008, p.558). Riggin (as cited in Tashakkori & Teddlie, 2003) also supports the use of mixed 
methods as it is “not only advisable but inevitable” (p.492) in the research design and collection of data. 

Fraenkel and Wallen (2008) classify mixed research designs into three basic categories: The exploratory design, 
which starts off with qualitative data, then quantitative, the explanatory design, which moves from quantitative 
to qualitative data and the triangulation design which studies both quantitative and qualitative data at the same 
time. Of these three, the one that applies to this particular study is the triangulation design: the data has been 
gathered from different parties involved in the computerized architecture, i.e. students in and out of the 
preparatory program, their instructors and the administrators as well as collecting data using multiple tools such 
as questionnaires with both numerical and a comments part, laboratory and class observations and interviews. 
The reason why mixed methods was used is that the quantitative and the qualitative data collected were 
expected to complement each other, thus providing a more reliable account of perceptions of those involved in 
the computerized tracks. 

Instruments 

The main instruments that were used to gather data were: 

(1) Student questionnaires for the group still studying in the preparatory program 

(2) Instructor questionnaires 

(3) Administrator questionnaires 

(4) Student interview forms for the group still studying in the preparatory program 

(5) Student interview forms for faculty students who passed the proficiency exam 

(6) Classroom observation forms for Track 2 

(7) Computer laboratory observation forms for Track 1 

Data collection 

Data collection spanned approximately four months. Initially the evaluation scheme was introduced to 
all parties involved, the observations started immediately. However, the researcher waited until week 6 of the 8 
week course so that the students would have completed most of Track 1 and 2. The interviews were carried out 
in week 8, which is the last week of the course, so that students would have completed Track 2. 

Prior to the actual research, the first version of the student questionnaires, consisting of 76 questions, 
were administered to a pilot group of 122 mainstream and repeat students, of which 60 were intermediate and 62 
upper-intermediate. The questions were filled in on optic forms, and they were sent for processing to the Student 
Services in the preparatory program, an administrative unit that conducts item analyses and are involved in item 
banking, who then transferred the results to Excel workbooks and offered their services for the analysis part as 
well. Inter-item correlation was also conducted on SPSS to see any overlaps between items. 

For reliability, Cronbach alpha was conducted for Part II, Familiarity with Computers; Part III, the 
scale section for materials in Track 1 ; Part IV, Reasons for Non-use, and Part V, Project Work in Track 2. All 
the factual questions were left out. Namely, these were the Part I, General Information and some sections of Part 
III, Materials in Track 1, which contained questions such as the number of materials completed or ordering from 
the most useful to the least. The reliability of all the subscales was examined, as well as for the whole 
questionnaire. Cronbach alpha values for each single part varied between .73 and .94. For the whole 
questionnaire, it was .94. Cronbach alpha was also computed for the real data from the instructor and the 
administrators’ questionnaires. For the instructor questionnaire, the values ranged between .73 and .90 for the 
separate parts. For the whole questionnaire, it was .91. For the HTU questionnaire, it ranged between .40 and .83 
for the separate parts. For the whole questionnaire, it was .85. 

Phase 1 

In Course 1 of the 2006-2007 academic year, an announcement was sent to teaching units explaining 
that an evaluation of Tracks 1 and 2 in both intermediate and upper-intermediate levels was going to take place 
during the course. The researcher herself also visited each intermediate and upper-intermediate teaching unit to 
explain the rationale behind the research, data collection procedures and also identified volunteer instructors for 
computer laboratory and classroom observations. 

Track 1 had been designed in concordance with the textbook; therefore, for each unit in the textbook 
that is to be covered in a week, there were corresponding materials in Track 1 for all of the skills and language 
components, except for speaking and writing. Each class in the intermediate and upper-intermediate levels had 
been allocated computer laboratory slots. Starting with week 3 of Course 1, classroom and computer laboratory 
observations started taking place. For two weeks the pre -determined intermediate and upper-intermediate 
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classes, 3 intermediate and 3 upper-intermediate, were observed, in the computer labs for Track 1. Each of these 
classes, with the exception of one, was observed three times in terms of 4 skills in Track 1 : Reading, listening, 
and grammar and vocabulary. Listening and reading lessons were observed in a 50- minute teaching block each; 
vocabulary and grammar lessons were observed in one 50-minute block as they were shorter than the skills 
materials. Due to different practices and preferences of the instructors, the number of classes observed was as 
follows: 

Table 2 Number of laboratory observations for each Track 1 skill 





Intermediate 


Upper-intermediate 


Only reading 


4 


1 


Only listening 


2 


3 


Listening or 
reading 


- 


2 


Grammar and 
vocabulary 


2 


3 



Due to a scheduling mishap, one reading lesson was missed in one of the Upper-intermediate classes. 
Instead a different class was used to observe another reading lesson. After each observation, the observation 
form was shared with the main class instructor and the instructor was asked if s/he shared the same opinion 
about the observation and to add his/her own comments if s/he had any. All of the instructors agreed with the 
written account of the observations. 

For Track 2 two classes from each level were also observed during oral presentations. The researcher 
made appointments with the main class instructors to observe the classes on presentation days for a 50-minute 
block. At the end, the class instructor was asked to comment on the filled in Classroom Observation Form and 
add his/her own comments if s/he liked. 

Phase 2 

In week 6, when both levels had finished the majority of Track 1 and 2, the questionnaires were 
administered to all of the intermediate and upper-intermediate classes. The questionnaires were filled in during 
class time under the supervision of the instructors. Each class was given an envelope which included the 
questionnaire booklets and optic forms which had options from A to E. The students were given half an hour to 
fill in the questionnaires. The instructions were explained to the instructors with a written memo as well as in 
person, and instructors were asked to explain them to students in class. 

The instructor questionnaires as well as the HTU questionnaires were administered in Week 7. A 
deadline was given to the instructors and the HTUs, and they were asked to return the filled in questionnaires to 
the researcher’s mail box so as to ensure anonymity. After the questionnaires were collected, the optic fonns 
were sent to Student Services, who processed them in the form of Excel workbooks. All the letters from A to E 
were also transformed into 1-5. The comments pages from the questionnaires were also transcribed to be 
evaluated with the interviews and the quantitative data from the questionnaires. 

In Weeks 7 and 8, intermediate and upper-intermediate students who were still in the program were 
selected through maximum variation sampling for interviews. From both intennediate and upper-intermediate 
levels, 12 classes were selected each. For the intennediate level, it was every fourth class; for upper- 
intennediate, it was every third. For the interviews the researcher tried to ensure that the number of the 
participants was representative enough through maximum variation sampling. This was done by selecting an 
equal number of males and females, mainstream students and repeaters, students scoring over 60% and those 
below 60%. In two of the classes, there weren’t any students that met the selection requirements, so the 
available students were selected. 

The students that are still in the program were invited to the researcher’s office, where they were asked 
questions from the interview form. In total 22 students were interviewed, 12 from intennediate and 10 from 
upper-intennediate. 2 students from upper-intennediate failed to show up. Each interview lasted around 25 
minutes. The interviews were conducted so as to have a more detailed insight into the students’ responses to the 
questionnaires as part of the triangulation efforts. 

Phase 3 

Faculty students who had completed Track 2 in either intennediate or upper-intermediate levels in the 
2005-2006 academic year were identified. The students were selected through criterion-based sampling: The 
researcher went through a list of freshmen students who had completed a Track 2 project either in Course 1 or 
Course 3 of the previous academic year. The researcher tried to choose students from different departments as 
well as different gender. The English instructors of these students were approached and were asked to infonn 
the relevant student of the interview. If the student consented, an appointment was made to meet these students 
after their English classes in an available classroom. In order to ensure full participation, the researcher herself 
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went to the appointment locations instead of inviting the students to her office as the classes and her office were 
on two different campuses. This group of students was approached after they had spent several weeks in their 
departments with the idea that they would have developed some familiarity with their departments and course 
requirements. This period was thought to be crucial in raising their awareness about what kind of tasks they 
would have to carry out in their departments. Thus, it was believed that they would be in a more objective 
position to able to evaluate the benefits of Track 2 projects on their learning behavior and transfer of learning to 
real contexts. 

Data Analysis 

In the study both quantitative and qualitative data were gathered through various instruments such as 
questionnaires, interviews and observations. The details of the data analysis are explained in the following 
sections. 

The Quantitative Part 

The quantitative data from the actual study were collected through preparatory student, instructor and 
HTU questionnaires, the analysis of which followed the same procedures described for the pilot study except for 
the inter-item correlation. Descriptive statistics were calculated with Excel and Cronbach alpha was calculated 
with SPSS for the reliability of all the questionnaires. Cronbach alpha levels for ranged from .76 to .93 for the 
student questionnaires. For the whole questionnaire it was .92. For the instructor questionnaire, the values 
ranged between .73 and .90. For the whole questionnaire, it was .91 . For the F1TU questionnaire, it ranged 
between .40 and .83. For the whole questionnaire, it was .85. 

The Qualitative Part 

The qualitative data were collected from three categories of instruments: the open-ended sections of the 
preparatory student, instructor and F1TU questionnaires, the classroom and laboratory observations, and 
interviews with both preparatory and faculty students. 

To increase reliability and objectivity as well as easing the coding process as suggested by Yildirim & 
Simsek (2000), the researcher quantified the open-ended sections of the questionnaires. These parts were 
analyzed by the researcher for Track 1 and Track 2 separately to find emerging themes under three broad 
categories for the sake of simplicity: positive comments, negative comments and suggestions/requests. Each of 
the comments for each track was then quantified and categorized. 

The laboratory observations were analyzed by the researcher under the headings of the Computer 
Laboratory Evaluation Forms: Students’ responses to the tasks assigned, students’ reactions to the program, 
students’ computer skills and students’ emerging needs. For reliability purposes these fonns were also checked 
by the class instructors after they were filled in. Common themes from the observations that were recorded on 
these sheets were identified and categorized as positive or negative. The observations led to very similar results 
so common themes emerged quite naturally. The classroom observations were recorded using the headings in 
the Classroom Observation Fonns: Students’ level of the grasp of the task assigned, students’ level of interest in 
the task they are involved in, classmates’ level of interest in their friends’ work and other comments. These 
fonns were also checked by the class instructors after the observations for reliability purposes. Similar to the 
laboratory observations, common patterns were categorized as positive and negative. Likewise, the themes 
emerged effortlessly from the classroom observations. 

The interviews with all the preparatory program students, both intermediate and upper-intennediate, 
and faculty students were transcribed into a 113-page document. The transcriptions were analyzed through 
content analysis. For the reliability of the coding process and finalizing the coding list, 4 interviews, 2 from 
intermediate and 2 from upper-intennediate, were initially coded by a second rater. First the total comments 
from both raters were calculated. Each of the codings was examined together with the second rater and the 
rationale behind the coding was discussed until reaching an agreement. Then, the initial coding list was finalized 
leaving out ambiguous codes, merging and reducing some codes. 

Results 

Track 1: Current Students in the Program 

It was found that the students completed Track 1 materials mostly after school and during computer 
laboratory blocks on a weekly basis or in their free time. The number of materials they completed was fewer 
than expected: approximately sixty percent of students had completed between 1-6 materials in individual 
strands. 

The Track 1 strand students most liked and found most useful was the listening strand. The one they 
liked the least and found least useful was the vocabulary strand, which can be explained by the fact that the 
vocabulary strand was not updated according to the latest word lists, so they included words that students were 
not tested on. The reading strand was the second best according to the upper-intennediate group, whose third 
choice was grammar. For the intennediate group, it was vice versa: grammar was their second best strand, 
whereas reading was ranked third or fourth. 

The students were able to see the connection between the course book and Track 1 strands and found 
them useful in learning English in general as well as for the during and end-of-course achievement tests. The 
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level was appropriate for most and the materials were relevant to their needs, which was also evident in the 
observations during which the majority of students were engaged with the materials. The materials were easy to 
use on their own. During the laboratory observations, there were no problems in terms of usage. However, it is 
hard to say the students enjoyed doing the materials. Nevertheless, they expressed their desire to see such 
exercises in other levels. One of the questions which had the greatest level of agreement was whether making 
Track 1 materials accessible through the Internet would increase usage. The complaints about the computer 
laboratories came from quite a few students, who felt the conditions needed to be improved. Although the 
students felt it was a good idea to learn English through the computers, they still preferred more traditional 
methods: using books, learning from the instructor and learning in class. 

Track 1: Instructors and Administrators 

As for the instructors and administrators most of them felt the training they received on Track 1 was 
adequate. However, only half of them had gone through the materials in detail themselves. The group was not 
very confident about making the best use of Track 1 materials. They felt they needed more time and access from 
home or the TU computers to familiarize themselves with the materials. They also wanted to be able to better 
track student usage of Track 1 materials. 

The vast majority was very positive about Track 1 materials. They said the connection between Track 1 
and the course books was clear, making Track 1 useful in learning English, and preparing students for the 
exams. The instructors and the administrators liked especially the listening strand like the students, which was 
followed by reading. They were not keen on the vocabulary or the grammar strands, which was consistent with 
the students’ preferences. They either encouraged their students to use the materials in the computer laboratory 
blocks or assigned them as part of the Outside the Class Strand (OCS) on a weekly basis. The instructors’ and 
administrators’ belief in the benefits of the materials is important in encouraging student engagement with the 
materials. According to the findings, most instructors and administrators supported the use of Track 1 materials 
for the way they provide practice outside the class. The vast majority said they would like to see such exercises 
in other levels as well. The general management support is also likely to be a factor in such acceptance of 
innovations in that the instructors are strongly encouraged to implement them in a systematic way. The need for 
more time and training for familiarizing with both Tracks also came out of the study. 

Track 2: Current Students in the Program 

The students did not enjoy doing the Track 2 project or using the electronic platform used to publish it 
and said they would not really like to do more projects in other levels. They did not have very positive feelings 
about the topics either. Despite this, more than half of them felt it was a good way of learning English. The 
students felt the projects helped learn transferable skills such as learning collaboratively and to a certain extent 
computer skills, and felt the projects could help them prepare for the department. They also expressed their 
interest in using computers and the platfonn used for Track 2 more in their studies. 

Track 2: Instructors and Administrators 

The instructors and the administrators had favorable views about the Track 2 projects. The vast 
majority believed the project effectively integrated all four skills and had the potential to change the teaching 
methodology in a positive way. They felt the students learned a lot about the topic, which made them think 
critically. They also enjoyed the project but as the students said the instructors did not think the students enjoyed 
it much. 

The instructors and administrators found Track 2 useful in teaching English as well as transferable 
skills. In addition, they thought the project was similar to what the students would do in their departments. They 
also stated that they would like to use computerized learning more in their teaching and design or help design 
such projects in the future. One request was to have more time to become familiar with the project, which was 
also made for Track 1, and have an earlier start during the course. This is very important as research indicates 
that the administration of the institution plays a very important role in the adoption of using computers in 
teaching (Dupaigne & Krendi, 1992). 

Track 2: Faculty Students 

Most of the students remembered their Track 2 experiences vividly, and compared to the current 
students had much more positive views about the benefits of the project. Although the project was seen as 
tedious, unnecessary and uninteresting by half of them at the time they were doing it, most of them realized its 
benefits for the skills they needed in their current department and their future careers. They specifically referred 
to the presentation, computer and research skills, taking responsibility, learning collaboratively and language 
learning as positive outcomes of the project. They also unanimously agreed that the project should be made 
available in other levels. 
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Conclusion 

The findings are explained under the headings of the two research questions The implications and suggestions 
resulting from the processes employed during the design, implementation and the evaluation of the particular 
computerized learning (CL) architecture studied include the following areas: 

• In tenns of student engagement: 

o Providing more incentives for the completion of the materials 
o Making computerized learning equally appealing and effective 
o Making the potential gains from CL more explicit 
o Adding the fun factor both to Track 1 and 2 
o Increasing student choice, thus voice, in Track 2 
o Providing learner training 

■ In usage of the programs 

■ In learner-centered instruction 

o Strengthening the role of CL as a core course requirement 
o Increasing accessibility 
o Improving the computer laboratories 

• In tenns of instructor engagement and ownership: 

o Providing more time for integrating CL 
o Ensuring a slower-paced implementation 
o Providing more varied and ongoing training 

o Continuing with strong management support during the implementation 
o Involvement of more instructors in the design and development 
o Diffusion of CL through best practices and change agents 

• In terms of instructional principles: 

o Placing the user at the center of the design process 
o Establishing a strong base of theories of instruction 

o Strengthening the close connection among all course components e.g. the syllabus, the 
textbook and CL 

o Strengthening the role of CL as a core course requirement 
o Continuing with the iterative design process to ensure quality check and control 
o Ensuring continuity in the implementation of CL 

• In terms of design principles: 

o Examination and selection of the software early in the design process 
o Ensuring the reliability and robustness of the software 
o Consideration of the needs and abilities of teacher-designer-authors 
o Adapting a team approach 

o Increasing the resemblance of the materials to the existing materials e.g. the textbook 
o Continuing with the iterative design process 
o Bringing in usability testing before launching CL 
o Continuing with modularity and easy adaptability of CL 

• In terms of research: 

o Establishing a team approach in the collection of data through mixed methods 
o Employing several methods in collecting data from students 
o Not aiming for generalizations 

This study investigated the general reactions of current and faculty students, instructors and HTUs 
towards the computerized learning architecture used in the preparatory program. The findings indicate that 
overall, all these groups have positive feelings towards architecture because of the way it is perceived as useful 
for both language learning and some transferable skills. The students in the preparatory program found the 
materials useful for their present needs as well as future goals although they didn’t really enjoy doing the 
materials, and still had a strong preference to learn in the traditional way rather than computerized learning 
packages. The instructors and the HTUs were on the whole very positive about both Track 1 and 2 in terms of 
their benefits for language teaching and transferable skills; however, a common request was to have more time 
to get familiar with and integrate the materials into their teaching. The students in their faculties were very 
positive about Track 2 projects in terms of transferable skills needed in their faculties and admitted that although 
they perceived them as a burden at the time they were studying in the preparatory program, they could see the 
benefits in their departments. 
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Abstract 

Expert instructional designers make sense of ill-structured problem situations more quickly than novices by drawing 
upon prior knowledge, previous experiences and an “established repertoire of heuristics” (Day & Lord, 1992, p. 45). 
This study examined heuristics used by expert instructional designers when engaged in the instructional design (ID) 
process. Qualitative interview methods were used to capture the individual and shared heuristics 16 instructional 
designers use when completing complex ID projects. Results are presented in the form of 8 categories and 25 
instructional design heuristics. Although some of the identified instructional design heuristics overlap with other 
fields of study such as business management and communication, others appear to be unique to this study. 
Implications for the field of instructional design are discussed. 

Solving problems is a big component of what instructional designers do (Muraida & Spector, 1993; Perez 
& Emery, 1995; Rowland, 1991; Seel, Eichenwald, & Penterman, 1995). They are presented with a situation that 
may be ill-defined and asked to close the gap between where we currently are and where we want to be. This can be 
considered a “problem.” When solving a problem, a person not only selects rules, but they combine multiple rules in 
order to arrive at a solution (Gagne, 1985). The solving of a newly encountered problem could require a learner’s 
use of declarative knowledge and cognitive strategies as well as the inclusion of previously learned relational and 
procedural rules. Johnson (1988) elaborated on the use of rules in solving ill-structured problems when he stated, 

“. . . no single correct procedure exists, and there is no definitive way of assessing the correctness of a rule based 
upon the outcome of a single case. There is no optimally correct rule. . . only rules which are relatively more 
accurate” (p. 212). 

This study is about expert use of heuristics in instructional design (ID). Heuristics are “simple decision 
rules through which individuals make judgments” (Dudczak, 1995, p. 4). When solving ill-structured problems, 
there is no one set of strategies that can be used in all situations, thus many people rely on heuristics to solve 
problems (Ingram, 1988). “A heuristic is a general strategy for approaching a problem” (Ingram, 1988, p. 215). 
Heuristics help to simplify the problem (Abel, 2003). Heuristics may be called many things: informal procedures, 
generalizations, strategies, principles, decision rules, intuition, or rules of thumb [list not inclusive]. For purposes of 
this paper, ‘heuristics’ and ‘rules of thumb’ will be used interchangeably (Abel, 2003). 

“Procedures for intelligently applying past knowledge to new experience often seem to require common 
sense and practical rules of thumb in addition to, or instead of, formal analysis” (Schank & Abelson, 1977, p. 3). 
According to Davenport and Pmsak (2000), rules of thumb are “guides to action” that have developed over time 
through extensive experience and observation (p. 10). Rules of thumb are “neither ‘simple facts’ nor general 
knowledge about reasoning, but are of the form ‘If you are trying to deduce this particular sort of thing under this 
particular set of conditions, then you should try the following strategies’” (Winograd, 1975, p. 190). 

Expert use of heuristics has been examined by a number of researchers (e.g., Day & Lord, 1992; Murphy, 
2005; Seel et ah, 1995; Woods, 1988). In a study by Day and Lord (1992), 38 experts and 30 novices participated in 
the task of sorting organizational problems. Results showed that experts categorized the ill-structured problems 
faster than novices. They specifically stated, “Experienced executives quickly make sense of relevant problems and 
draw upon an established repertoire of heuristics to guide their sense-making efficiently” (p. 45). Chi (2006) stated 
that experts may use heuristics of which novices are not aware. Although, experts may perform with some heuristics 
so automatically, that even the expert is unaware of their use (Lee & Reigeluth, 2003). 

The foundational piece on heuristics in instructional design is a piece by Haney, Lange, and Barson (1968). 
The premise was to provide instructional developers with heuristics to use when working with professors in higher 
education. From empirical observations of an instructional development project, they created a collection of 18 
heuristics. These heuristics are so general that even though they were compiled 40 years ago, they still apply today. 
Hoban, Heider, and Stoner (1980) elaborated on Haney et al. (1968) by identifying additional heuristics they noted 
from their empirical observations of a large instructional development project. Noel and Hewlett’s (1981) theoretical 
article on heuristics in instructional development derived from their own experiences and that of colleagues. They 
explained when heuristics can be used in the development process. They listed both general heuristics as well as 
heuristics that align with the instructional systems development (ISD) model. 
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Ertmer et al. (2008) suggested expert instructional designers use their prior knowledge and experiences to 
create and apply personal “rules of thumb” (i.e., heuristics) when solving instructional design problems (Haack & 
Mischke, 2005; Kolodner, 1997; Rowland, 1993). These rules are not taken from textbooks, but are much more 
personalized - acquired from the individual’s schema of previous experiences that they bring to the design situation 
(Klein & Calderwood, 1988; Kolodner & Guzdial, 1999). In his textbook. Designing Instructional Systems, 
Romiszowski (1981) conjectured that instructional designers take the systematic approach they were taught and 
change it into heuristics they can use. 

Jonassen and Hernandez-Serrano (2002) addressed the complexity and ill-structured aspect of instructional 
design problems in that they offer much uncertainty during problem-solving, thus experts tend to use more heuristics 
than if the problem were well-structured: “When reasoning under uncertainty, people rely on a variety of heuristics, 
that is, rules of thumb” (Kunda, 1999, p. 109). Nelson (2003) stated, “designing, like problem solving, is based on 
systematic processes and situational ‘rules of thumb’ (Perez, 1995) that should lead to purposeful and practical 
outcomes” (p. 40). The literature supports the use of heuristics in instructional design (Haney et al., 1968; Hoban et 
al., 1980; Noel & Hewlett, 1981; Silber, 2007; van der Meij & Carroll, 1995) however it is time for some empirical- 
based evidence of heuristic use. 

Purpose 

This study was designed to explore what heuristics experienced instructional designers use when engaged 
in the instructional design process. Qualitative methods were used to capture and describe the individual and shared 
heuristics instructional designers use from the instructional designers’ perspectives. In order to achieve this, semi- 
structured interviews took place with instructional designers who have practiced instructional design for at least ten 
years. Instructional designers were purposively selected (Patton, 2002) according to their experience with 
instructional design as well as their availability to be interviewed. During this interview, participants described their 
experience in the form of a narrative. These instructional designers reflected on complex or challenging instructional 
design projects they have worked on in order for us to elicit heuristics they now use in their practice. Specifically, 
the research question was: What heuristics do experienced instructional designers use to solve instructional design 
problems? 

Theoretical Framework 

A qualitative phenomenological research design, utilizing semi-structured interviews (Patton, 2002) of 
expert instructional designers, was used. Phenomenology originated as a method of philosophy based on the 
investigation of phenomena in the 20th century by German philosopher Edmund Husserl (Smith, 2005). 
Phenomenology is an approach that revisits the conscious experience as a way of capturing the core of that 
experience through thorough first person descriptions (Moustakas, 1994; Smith, 2005). Interviews elicited this first- 
person point of view and situated it within the participants’ narratives of their instructional design experiences. In 
order to develop a deep understanding of the essence of instructional design heuristics by experienced instructional 
designers, it was important to examine a group of instructional designers’ experiences (Creswell, 2007). In this 
study, the researchers elicited multiple participants’ personal experiences of instructional design through their own 
words. This framework enabled the researchers to elicit first-person points of view and situate them within the 
instructional design experience in order to get an accurate picture of heuristics used in actual (not theoretical) 
instructional design projects. The researchers’ intent was to determine what heuristics instructional designers 
perceived they used in their past instructional design experiences. In order to understand the heuristics described by 
the participants, the context in which they were used needed to be described by the participants as well. This context 
formed the essence of the experience surrounding the heuristic use. Because the participants share the experience of 
being instructional designers, there is a component of phenomenology that allows for the examination of the 
“essence of shared experience” (Patton, 2002, p. 106). Therefore, in the attempt to elicit instructional designers’ 
shared heuristics from their past experiences, phenomenology was the approach that was undertaken for this 
research. 

Methodology 

Participants and Site 

A purposeful sampling technique (Creswell, 2007) was used to select potential participants who had at least 
10 years of instructional design experience, are currently active in the field, and known by their peers as expert 
instructional designers. An initial list of potential participants (n = 31) was created from a list of “seasoned” 
professionals who regularly attend the annual Association for Educational Communications and Technology 
(AECT) conference. This was done by looking at past conference programs and identifying those who the consulting 
professor had professional knowledge of and thought were both experienced enough and might be willing to 
participate. In addition, participants were selected based upon their availability for either a face-to-face interview at 
the Association for Educational Communications and Technology (AECT) 2007 conference or a phone or video 
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conference in the three months following AECT. The snowball method (participants identifying other potential 
participants) was used to increase the number of potential participants to 49. After approval from the university’s 
IRB, persons who met the inclusion criteria were emailed requesting participation. Participants (n=16) had an 
average of 23 years of professional instructional design experience. There were six female and ten male participants. 

Of the 16 participants, 7 (44%) were interviewed at AECT face-to-face while 9 (56%) were interviewed in 
the following months via videoconference and phone interviews. In addition, those who agreed to participate were 
asked to complete a demographic survey that allowed the research team to determine if they met a minimum level of 
criteria for expertise in order to be included. For example, 10 years of professional instructional design experience 
(Ericsson & Charness, 1994; Ericsson & Lehmann, 1996; Ertmer et al, 2008; Eseryel, 2006; Foley & Flart, 1992; 
Korth, 1997; LeMaistre, 1998; Perez, Johnson, & Emery, 1995; Perez & Neiderman, 1992; Rowland, 1992), highest 
educational degree (Day & Lord, 1992; Ericsson & Charness, 1994; Eseryel, 2006; Korth, 1997), and whether they 
are currently practicing/managing instructional design (Ericsson & Charness, 1994; Ertmer et al, 2008; Rowland, 
1992). All 16 participants met the aforementioned minimum criteria for expertise as defined for this study. The 
research team felt 16 interviews would be enough to answer the research question, but left open the option that if 
after the interviews they felt they needed more data, more participants could be found and more interviews 
conducted. 

Role of the Researchers 

The first author took primary responsibility for the research, including designing and implementing the 
study. A faculty member and a visiting scholar acted as consultants. The first author is a doctoral student in an 
educational technology program at a large mid-western university. With each participant she established her own 
background in instructional design. This was done to foster their freedom to use instructional design terminology 
during the interview. However, it also led some of them to take on a “teacher” perspective since she was considered 
a “novice” or “student.” Whether this impacted their responses during the interview or not, cannot be said. 

Interview protocol 

A six-page interview protocol was created and piloted by the research team. It included the steps the 
researchers should complete before, during, and after the interview, as well as demographic survey questions, 
research questions, possible interviewee questions and appropriate responses, follow-up questions, and an interview 
check-out list (to ensure necessary data were collected). Jonassen and Hernandez-Serrano (2002) stated that 
experienced practitioner stories provide relevant information about strategies they use in the instructional design 
problem-solving process. They listed specific steps for working with practitioners to identify lessons learned. Some 
of these steps were used in the interview protocol as probing questions. For example, participants were asked to 
identify the problem goals and to describe the context in which the problem occurred. In addition, they were asked 
to describe the solution that was chosen and the outcome of that solution. Participants were also asked to identify 
any lessons learned from the project. 

Prior to the start of the study, the research team conducted three practice interviews with professors in the 
Educational Technology program at the university, in order for the lead researcher to familiarize herself with asking 
the interview protocol questions. Subsequently, the research team refined the interview protocol; some research 
questions were added, eliminated, and altered for better interview communication procedures. 

Demographic survey 

A demographic survey was provided via email to participants prior to interviews. It requested the following 
information: participants’ gender, age, current position, number of years in that position, current organization or 
institution, highest degree earned, area of specialization, number of years in the instructional design field, context 
and years of instructional design positions, and instructional delivery formats used. 

Data Collection 

After approval from the university’s IRB, persons who met the inclusion criteria were emailed requesting 
participation. Once participation was accepted, the demographic survey and consent form were emailed to 
participants. Participants returned the survey and consent form via US Postal Service, Email, in-person, or fax. 

Interviews were conducted over a four-month period. Interviews took place in a face-to-face manner at 
AECT, as well as via telephone and video-conferences; at a time and location convenient to participants. An email 
was sent to participants verifying interview location and time as well as providing them a brief question to prepare 
them for the interview. A semi-structured interview was used to elicit a story about a complex or challenging 
instructional design problem on which the instructional designer has worked. In-depth information was obtained 
from each participant in the form of personal narratives. Interviews were used to elicit these personal narratives so as 
to acquire the participants’ perspectives using their own words (Kvale, 1996). The lead researcher began the 
interview by stating the following: 
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I would like you to start by telling me a story about an instructional design project you worked on in which 
you had to solve a complex or challenging instructional design problem. If you could, I would like you to 
start with you entering the situation and continue the story until you solved the problem. Include as much of 
the following as is needed to make the story understandable: setting, context, environment; characters 
involved, roles; time span or timeline; and actions, events, or decisions. I am interested in hearing your 
story about solving the instructional design problem. Remember the ultimate goal is to identify some of the 
rules of thumb you use in your practice. 

Narratives were elicited via interview questions such as: (a) What major constraints or functional requirements 
existed for the completion of the task? (b) How did you go about determining possible solutions? (c) What was the 
outcome of that solution? and (d) Were there any lessons from this ID problem you’ve continued to utilize in solving 
future ID problems? Probing questions were used by the lead researcher to elicit lessons learned and extract rules of 
thumb by which the participant operated in the design process (e.g.. In hindsight, were there any aspects of the 
project that could have been improved? What types of lessons did you take away from this ID problem? After 
having these experiences, what advice would you give to someone who has just discovered that he or she is in the 
same situation? What is your greatest challenge when trying to solve ID problems?). Each interview lasted between 
30 and 90 minutes. All interviews were audio/video-taped and transcribed verbatim. 

Transcription of the interview data was completed with outside assistance. The lead researcher verified the 
accuracy of each transcript by reading through the transcript while listening and viewing the audio and videotaped 
interviews. Corrections to transcripts were made during the verification process. 

Data Analysis 

Phenomenological methods employed by Colaizzi (1978) and modified by Moustakas (1994) were used to 
analyze the data. Giorgi (1985) described the two separate descriptive levels of the approach: Level one utilizes the 
original data (the verbatim transcript of the interview) and can be considered ‘naive’ descriptions, whereas level two 
contains the researcher’s interpretation of the data, where through reflective analysis, he/she describes the structures 
of the experience. These structures are the “underlying structures that account for an experience being what it is” 
(Moustakas, 1994, p. 137) and are the ‘essence’ of the experience. In the final step, the researchers combine the two 
descriptions (raw data and inteipreted meanings) into a comprehensive analysis of the phenomenon. For example, a 
level one comment was, “1 let people describe the problem. And again, even though it can be difficult for me, I will 
not ask any questions, what so ever, until they are completely finished.” This statement is verbatim from the 
interview. The level two analysis led to, “the instructional designer needs to really listen to the client when the client 
is describing the problem”. From these two descriptions, the rule of thumb that emerged was, “As a designer you 
need to listen more than you talk.” 

To begin, videotapes were viewed and audiotapes listened to as well as transcripts read a number of times 
in order to obtain an overall feeling for the interviews. Significant statements related to instructional design 
heuristics were identified in the transcripts. These were highlighted and extracted. Statements were written on index 
cards for easier organization. Systematic steps (i.e., open coding, axial coding, and selective coding) were used to 
generate categories of rules of thumb (Creswell, 2007). Statements were then grouped into themes (within the 
categories) common to all participants’ transcripts. A theme would be a recurring idea, thread, meaning, etc. that 
might not be mutually exclusive; statements might fit into more than one theme (Graneheim & Lundman, 2004). 
Themes were then described as an in-depth look at the phenomenon. Findings were validated with the participants 
and any discrepancies were negotiated or noted (Creswell, 2007). 

Trustworthiness of the Study 

Trustworthiness of the study appears through the systematic, rigorous steps executed in the data collection 
and analysis procedures (Patton, 2002). The development of an interview protocol and use of a pilot study ensured 
consistency of the interviews. The interview process allowed for the collection of rich, detailed, personal narratives 
from the participants. Reliability of the data was enhanced by viewing the videotapes multiple times, listening to the 
audiotapes multiple times, and verification of the transcripts by a second person (Patton, 2002). Through the 
numerous readings of the transcripts, along with listening and viewing audio and videotapes, the lead researcher was 
able to bracket out her own personal experience and present only the participants’ perspectives (Creswell, 2007). 

Results 

This study examined the heuristics used by experienced instructional designers when solving complex 
design problems. Analyses of interview transcripts led to the identification of eight categories of heuristics. 
Categories included: (1) communication, (2) management, (3) learner/audience, (4) solutions/deliverables/outcomes, 
(5) design process, (6) design team, (7) design problem, and (8) client. Within each category, multiple heuristics 
were identified. In the next section, we describe the eight categories, the heuristics included in each category, and 
supporting statements made during the interviews by participants. It should be noted that these are not listed in any 
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order of importance. In addition, the number of participants who referred to a specific heuristic is listed after the 
heuristic. 

Communication 

The Communication category (see Table 1) includes heuristics that relate to communication between the 
instructional designer and his/her client. In this category six participants discussed listening to what the client was 
saying in order to get as much information as possible. For example, “Go to the meeting and keep your ears open. 
They are going to say things there that you are going to need to know about later” [E.B.]. 

Once the instructional designer felt he/she had the information from the client, five participants said it was 
necessary to verify that he/she heard that information correctly. For example, “The first thing I want to do is I verify 
things that I’ve heard. Just literally, honestly, and completely verify the things that I’ve heard. Not to ask a leading 
question, but just simply for me to verify that what I think I’ve heard is what the client is trying to say” [J.Q.]. “I put 
things in writing and I say ‘do you agree with this? Is this accurate? Let’s change this until it’s accurate’” [S.P.] . The 
confirmation message, written after meetings and sent usually by email, seemed to be an additional way of 
documenting and verifying statements regarding the instructional design problem. Seven participants discussed the 
importance of visuals and documents when communicating with the client. That documentation was very important 
in facilitating communication. As one participant described, “It is always more successful for me to show people 
what I am thinking, than it is to tell them about it” [E.B.]. Another said, “Visually portray whatever process model 
you will be using and consider sharing that with the client or the primary stakeholder group early on in the process 
to help make sure the lines of communication are clear” [R.B.]. 

In addition, three participants discussed using the correct lexicon when conversing with the client in order 
to facilitate communication and reduce misunderstandings. “I think the important thing for any designer to do would 
be to provide yourself with ample time to learn the lexicon ahead of time so that you can more easily just follow on 
the conversations faster” [P.L.]. This means the instructional designer needs to understand the terminology used by 
the client and only use instructional design terminology if it is necessary to the conversation. Otherwise, it was 
suggested that the instructional designer use terms the client would understand. As J.Q. explained, “We use our 
technical terms, but when we talk to clients, not only should we not do it unless we explain it, I don’t think we 
should do it unless we need to do it. It builds up a barrier in a sense, or it can, with a client if they aren’t familiar 
with this terminology.” 



Table 1 

Communication Instructional Design Heuristics 

Heuristic No. Participants 

As a designer you need to listen more than you talk. 6/16 

Verify all information received from the client to prevent miscommunication. Send a follow 5/16 

up email (confirmation message) after meetings. 

Use visuals and documents to prevent miscommunication. 7/16 

The instructional designer and client need to understand the lexicon used by the other. 3/16 



Management 

Managing the first meeting with the client was discussed as important by three participants (see Table 2). 
One participant explained how the first meeting progresses: “I would say to [the client], ‘In the first meeting, you are 
going to be doing a lot of the showing and talking because I want to hear all your perspectives on this situation that 
you are faced with. I will be taking notes and if you have any materials, etc. send them to me ahead of time.’ And 
then I will say, ‘I will either give you some initial thoughts on the situation, or if the problem seems quite complex, I 
will take it away and come back in a week and give you my thoughts on where we should go from then on’” [J.Q.]. 
This indicates that the instructional designer took time to process the provided information and did not just jump 
into designing in the first meeting with the client. R.B. supported this notion when he said, “Don’t even do any 
instructional design things in the first meeting.” 

In addition to managing the first meeting with the client, five participants stated how the instructional 
designer needs to identify the constraints placed on a project as well as negotiate the scope of the project with the 
client. “The first thing I do is 1 look for the constraints that have been placed on the project. Those constraints are 
the key to design” [A.G.]. M.G. stated, “It was really ingrained, very early in my head, about how limitations at the 
end impact what you are able to do at the beginning, and so understanding what those limitations are really makes it 
more successful for you to not have to redo work.” When looking at the beginning of a design project and the 
constraints, eight participants discussed how a statement of work should be created and verified by the client. As 
M.Sp. noted, “Getting the scope of the effort right is important. And I think that people are constantly 
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underestimating what the scale of the training design effort was going to be, they just never scoped it correctly. They 
underestimated how long it would take, how much and what kind of resources would be required.” Because of the 
constraints placed on a project by the client, as well as the resources provided, four participants discussed not being 
able to create “perfect” instructional design. As J.Q. explained, “One cannot often do the best project that you would 
like to. You can do the best instructional design with the resources that your client provides.” 



Table 2 

Management Instructional Design Heuristics 

Heuristic No. Participants 

Don’t do any instructional design in the first meeting. 3/16 

Look for constraints that have been placed on a project. 5/16 

Negotiate the scope of the project. Create a statement of work upfront. 8/16 

Do the best ID you can with the given resources and constraints. 4/16 



Design Team 

This category resulted from four participants describing how important it was to have the right people on 
the team (see Table 3). “Design is a people process. If we don’t have tools, if we don’t train ourselves to think about 
how we work with people and how we respect people, especially including our coworkers, that has a huge impact on 
what we are a able to do and how things get done” [E.B.]. 

Some participants discussed involving everyone from the beginning, while others stated that they only 
involve certain people at certain times. As E.B. stated, “One of the things they did that really impressed me was, we 
didn’t just sit with one writer, he called in the whole team. The entire group came. We didn’t even know why some 
of them might be necessary. . . but that rule of thumb goes something like this: Get together all the excited, exciting 
people you can find and when the problem is truly something unique to them bring in more, not less and don’t ask 
yourself, ‘well, are we really sure we need one of these or one of those?”’ Whereas W.C. noted that more than three 
people can stifle the process, “One person is almost never as smart as two. Two people are never as smart as three, 
but when you begin to get above that, it gets really interesting. You’ll get lots of ideas, but you’ll have a hard time 
getting finished.” Those who discussed this category agreed that instructional design is done in teams and that you 
have to work well with other people in order to come to a viable solution. “I would say that one of the common 
problems that you see is, especially if you’re working with a team, that it’s hard to get internal consistency. It’s hard 
to get everybody to really communicate enough so that you finally come to an agreement on what it is that you’re 
doing. That’s the real problem . . .” [S.P.]. 

Table 3 

Design Team Instructional Design Heuristics 

Heuristic No, Participants 

Design is a people process. 4/16 

The team is critical. Involve the right people at the right time. 4/16 



Design Process 

The Design Process category (see Table 4) included items related to the overall instructional design process 
such as knowing the foundational theories, design models, and past solutions. Six participants mentioned that 
analysis was a very important step in the instructional design process. As N.D. stated, “Analysis, analysis, analysis; 
you can never do enough analysis.” M.G. also stated, “Making sure that as you move through the instructional 
design process that you come back to the analysis component again and again.” 

In addition, understanding the fundamentals of learning theories, instructional design models, and 
principles of instruction seemed a predominant theme discussed by six participants throughout the interviews. For 
example, M.T. said, “You need to know the theories. You need to know the models. You need to have that 
foundation. If you don’t have that foundation. . .that almost needs to be your ground floor.” D.M. noted, “We need 
to implement those principles of instruction that work.” 
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Table 4 

Design Process Instructional Design Heuristics 

Heuristic No. Participants 

During the ID process revisit the analysis component again and again. 6/16 

You need to know the theories and models. Implement those principles of instruction that 6/16 

work. 

When faced with something complex, draw from past experiences. 5/16 

Another concept discussed by five participants in the Design Process category was the idea of using ideas 
that have worked in the past. For example, W.C. stated, “Don’t try to break too much new ground.” A.G. expanded 
on this by explaining that you can use past situations that have worked, but to add to them, “Be prepared to have to 
think abstractly, and don’t just copy old solutions, try to add something new in every design.” 

Learner /Audience 

The Learner/Audience category (see Table 5) was mainly comprised of knowing your target audience and 
what it was you wanted them to be able to do after the instructional experience. This specific heuristic had the 
largest number of participants (n=8) discuss it. As W.C. stated, “Invest as much time as you can in your audience 
analysis because it pays probably the biggest returns of all of them. Task analysis gets a lot of attention because it is 
core to it, but a lot of times people don’t spend enough time looking at learning skills and other things about the 
audience and if you’re not careful you’ll miss some crucial property of that audience in terms of their experience.” 

In regards to the performance outcomes of the learners, seven participants discussed knowing what it was 
you wanted your learners to be able to do at the end of the instructional experience. “When I approach any 
instructional design problem, 1 always try to use the mantra of, ‘what do we want people to be able to do?’ How can 
you create an instructional activity that approximates the kinds of activities they are going to be doing in the job you 
are trying to prepare them for?” [P.P.]. B.B. stated, “Let’s spend our time giving [learners] opportunities where we 
show them what success looks like. Give them an opportunity to practice it. And give them feedback about how to 
improve. Teach them how to do it well, give them practices to show they can do it well and they’re going to be 
motivated.” This indicates that not only should an instructional designer look at the performance outcomes at the 
end of the instructional experience, but also look at the desired performances throughout the experience. 

Interestingly, it appears that not all of the designers agreed about the specific implementation of the rules. 
For example, in the Learner/ Audience category, although four participants discussed including scaffolding in the 
instructional experience, as shown in the heuristic “Consider utilizing scaffolding in your instructional experience,” 
S.P. felt it was necessary to hold off on using scaffolding in the beginning whereas B.W. thought it should be used in 
the beginning and then tapered off at the end of the instruction. “Don’t scaffold at first. Don’t direct. In the first 
version you want to give the minimal amount of direction and support that you can give - because you really want to 
find out what it is the student needs. Where things break down. And then start trying some of those things out on a 
verbal basis” [S.P.]. “Use relatively high scaffolding and support early with moving toward more independent work 
as they get their bearings” [B.W.]. These different viewpoints both take into consideration using scaffolding, but the 
amount and placement vary depending on the instructional situation. 

Table 5 

Learner/Audience Instructional Design Heuristics 

Heuristic 

Know your learners/target audience and their pre-requisite knowledge. 

Determine the criterion for successful performance. 

Give learners the tools they need to succeed. Consider using scaffolding in the instructional 
experience. 

Solutions/Deliverables/Outcomes 

The Solutions/Deliverables/Outcomes category (see Table 6) has three heuristics. Four participants 
described the first, whether instruction is the answer to the client’s problem. “It’s really important that you recognize 
that education and training will only get you so far, education and training only are appropriate viable solutions 
when there is a skill and knowledge deficit” [B.B.]. Or as J.Q. stated, “I always listen quietly and I’m always asking 
myself, ‘is it actually instruction that is needed here or not’.” 

The second idea was discussed by eight participants (again, the largest number of participants to discuss a 
heuristic). This rule is to approach the design problem with the end in mind. This entails considering what the 
deliverables, learning, and performance outcomes are. “When it comes to instructional design, I identify what the 
deliverables are, what do we want to be able to accomplish at the conclusion of the project. And then work 



No. Participants 

8716 

7/16 

4/16 
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backwards from there” [M.Si.]. This heuristic is very similar to one in the Learner/ Audience/Pedagogy category: 
Determine what it is you want your learners to perform after the instructional experience. However, it differs slightly 
in that this heuristic is the way to approach the problem, not just a step in the process. As B.B. stated, “Always start 
with figuring out what are the ways you want people to change as a result from this experience.” 

In addition, four participants suggested generating and providing the client with multiple solutions to 
solving the problem and letting the client choose which one to implement. For example, “We got to the point where, 
after developing instructional strategies, we developed alternatives for several instructional strategies so that even 
the clients could make some judgments about which might be best or which might be preferred given the 
circumstances of that specific situation” [R.B.]. “You have to first understand a variety of solutions, so you have to 
have an idea of what are all of my possibilities of outcome” [P.L.]. 

Table 6 

Solutions/Deliverables/Outcomes Instructional Design Heuristics 

Heuristic No. Participants 

Ask yourself, “Is instruction the solution to this problem?” 4/16 

Approach the design problem with the end in mind. What are the deliverables? What are the 8/16 

learning/performance outcomes? 

Generate multiple possible solutions that will solve the problem. 4/16 



Design Problem 

The Design Problem category (see Table 7) comprised understanding the particular unique design problem. 
This category had the fewest participants discussing the same concept, but the researchers felt these heuristics were 
important enough to include. Two participants mentioned that although the instructional designer might have 
worked on similar problems, every situation is unique. As B.B. noted, “I think because every single situation is 
different, because every single situation is unique, that taking clients language to figure out how can I turn this into a 
viable solution that actually addresses the need and I am not just doing something to do it and I’m not just doing 
something they asked to do.” E.B. said, “You find out immediately there are no cut and dried situations, there are no 
situations where you can look up in your book [acts as if following a diagram] right. . .this is what we need to do.” 

In addition, three participants said that getting to the core of the problem and identifying all of the sub 
problems was important. For example, “I came back wondering - have I got the real problem? Have they just sort of, 
for whatever reason, just sort of given me pieces of the problem because of things that might be going on in their 
own organization or for their own needs, or am 1 forgetting to ask something about the problem that is really 
important, but I didn’t think to ask?” [J.Q.]. A.G. stated, “I think that every design problem that comes to us has 
many sub problems that need to be solved. If we can look at our design in terms of, what I call layers, of the design, 
we find that we solve a bunch of individual problems which then integrate together into one coherent design.” 

Table 7 

Design Problem Instructional Design Heuristics 

Heuristic No. Participants 

Never look at the problem at face value. Get to the core of the problem and solve all the sub- 3/16 

problems. 

Understand that every design situation is unique. 2/16 



Client 

In the Client category (see Table 8) six participants discussed determining if the client was able to describe 
what it was they really wanted from the instructional designer. And once this was communicated, the instructional 
designer needed to “manage” the client’s expectations. Part of this overlaps with developing a statement or scope of 
work. As N.D. stated, “Manage the client’s expectation. So don’t just take the project as is. Look at the project, look 
at what they want you to develop, look at your time limits.” As M.Si. said, “Many times the client comes to you and 
they’re not really sure what they want. And so you run the risk of proposing something that will satisfy their initial 
need for some help, some assistance, but without really solving the instructional design situation that brought them 
to you in the first place.” In order to determine what the client wants, the instructional designer needs to listen. As, 
P.L. stated, you “understand the client by listening to them. By listening to what questions they ask you.” 

Another theme that emerged from the Client category was that different clients have different levels of 
cultural sensitivity. Five participants discussed cultural and contextual sensitivity when working with a client. “You 
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have to be sensitive to the context. It’s probably the biggest thing that will help you be successful or not” [P.L.]. By 
creating a trusting relationship utilizing honesty, the instructional designer and client will have a more successful 
partnership. For example, a project that R.B. discussed was centered on tax accountants who worked with huge sums 
of money. This resulted in the instructional designers maintaining sensitivity when asking about “the client’s 
companies, what were their sources of income or what were their expenditures” as well as “how questions are 
perceived by the client.” Or as B.B. stated, “I tell them why I’m doing what I’m doing, think out loud, try and help 
them understand I’ve got their best interest at heart and go from there. But that means instructional designers really 
do need to have principles and practices that guide them.” The trust and honesty components of working with a 
client go hand in hand. As M.T. phrased it, “Be honest with the client- in terms of your recommendations.” M.Sp. 
said, “I think you have to be totally brutally honest with the clients.” 



Table 8 

Client Instructional Design Heuristics 

Heuristic No. Participants 

Determine if the client knows what he/she wants. Manage the client’s expectations. 6/16 

Be sensitive to the context and the culture of the client. 5/16 

Build trust with clients so that they know you have their best interests at heart. 3/16 

Be honest with the client. 3/16 



One interesting theme from the results was how many times participants would state that “ Blank is the key 
to instructional design.” For example: A.G. stated that constraints are the key to design. “The first thing I do is 1 look 
for the constraints that have been placed on the project. Those constraints are the key to design.” Whereas M.T. 
thought the key to design was recruiting the right people. “. . .and that was key- recruiting the right people.” M.T. 
also thought that asking the right questions of the client as well as looking at the problem from different perspectives 
was also key. “Asking the right questions is critical. And looking at it from many different perspectives. Those are 
really key.” B.B. felt that understanding what performances the learner should do at the end of the instructional 
experience was key. “If I can articulate what the practice experiences should be and what the critical success factors 
are for knowing how somebody’s performed in that practice what success looks like or doesn’t look like, and if I can 
figure out what demonstrations, if I can get that in place, then I’ve done what 1 think is the key job of an 
instructional designer.” This demonstrates that there are many aspects to instructional design that cannot be ignored. 
It also begs the question, are some rules of thumb more important to the instructional design process than others? 

Discussion 

Results from this study suggest that instructional designers use a number of heuristics in their ID practice. 
Ertmer et al. (2008) also described the use of rules by instructional designers in their study examining how expert 
instructional designers used knowledge and experience in the instructional design process. The twenty-five 
heuristics listed above were found as themes discussed across interviews. The list of heuristics above is a partial list 
due to the limited number of participants. Some of these heuristics seem to be common sense, such as verifying 
information to avoid miscommunication. Three of the eight categories of heuristics (Communication, Client, and 
Management) would appear to be common knowledge to business, or project management fields. One possible 
reason for the overlap would be the common use of problem solving in the different fields. In the management 
science / organizational research (MS/OR) field, problem solving is a common component (Evans, 1992). He stated 
that many MS/OR problems fall into the ill-structured category. This is the same with instructional design (Jonassen 
& Hernandez-Serrano, 2002). Powell, (1995) described three heuristics used in the MS/OR field that are similar to 
the heuristics found in this study. For example, one of the heuristics he describes is “Decomposition” of the 
problem, or in other words, break down the problem into smaller problems. This is similar to the ID heuristic we 
found: “You have to get to the core of the problem and solve all of the sub-problems.” It has been said that breaking 
down the problem into smaller problems allows it to be solved easier (Goel & Pirolli, 1988; Jonassen, 1997). In 
Nelson’s (1988) study, experienced instructional designers did just that and broke down the larger ill-structured 
problem in order to work on sub-problems separately. 

Another MS/OR heuristic Powell (1995) noted was “Prototype: Get Something Working.” This is similar to 
the ID heuristic we found: “When communicating with the client use visuals and documents.” Two of the 
participants in this study did mention using prototypes when involved in instructional design, but not as a rule of 
thumb they follow. More in common with this ID heuristic is Powell’s “Sketch a graph: Visualize” heuristic which 
is a manner of representing the system they are working on. Shanteau (1992) found that experts use decision aids or 
visuals when working on a problem that requires decision-making. In Hartley and Bmckmann’s (2002) book, 
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Business Communication, they discuss how to use visual aids as a way to reduce miscommunication. These all 
support the use of visuals and documents when communicating with a client. 

Other heuristics could have been predicted from the instructional design literature and the fact that the 
majority of the participants are in the academic field of instructional design. Those would consist of the Design 
Process heuristics, the Learner/ Audience heuristics, and the Solutions/Deliverables/Outcomes heuristics as many of 
these are taught in introductory instructional design courses and are in the textbooks. For example, using the models 
and theories, knowing your target audience, and determining learning outcomes is in a number of instructional 
design textbooks (e.g., Dick & Carey, 1985; Reiser & Dempsey, 2002; Romiszowski, 1981; Smith & Ragan, 1999). 
Of the instructional design heuristics above, the lead researcher has yet to find the following in the textbooks or 
journal articles: (a) As a designer you need to listen more than you talk, (b) Don’t do any instructional design in the 
first meeting, and (c) Generate multiple possible solutions that will solve the problem. This last heuristic seems to 
follow the idea that ill-structured problems have multiple solutions (Jonassen, 2000). Flowever, the idea of 
presenting the client with alternatives seemed to be the supporting foundation for this heuristic. 

What is unique about the list is that it combines these different areas of business, project management, and 
design into one overall package of heuristics that instructional designers might possess. This leads us to ask if there 
should be a business/project management course in instructional design curricula. 

Not every participant in this study described using every heuristic. This does not necessarily mean that they 
do not use that heuristic, only that it did not emerge during the narrative interview. It could be because the 
interviews were limited to one story told by participants, although a few participants told more than one story 
because the context of their story was not complex enough to answer the interviewer’s questions. 

Educational or scientific importance of the study 

Why is it important to have a list of heuristics used by practicing instructional designers? Some believe that 
because instructional design is a problem-solving process, novice instructional designers need to be taught more of 
what practicing instructional designers actually do, rather than only teaching instructional design model steps 
(Rowland, 1991; Silber, 2007). Allowing novice instructional designers to learn vicariously through the stories told 
by practicing instructional designers may provide them experiences they would otherwise not encounter (Jonassen & 
Hernandez-Serrano, 2002). If we can determine the heuristics practicing instructional designers believe should be 
taught to novice instructional designers, the field of instructional design could gain an additional tool to use in the 
education of novices as well as to help increase the foundational knowledge of the field of instructional design. 
Having a list of heuristics may help make designers more efficient in their design problem-solving processes. 

The underlying rationale for examining expert use of heuristics is to help novice instructional designers 
develop expertise. However, this concept needs to be examined in future research. It is currently not yet known if 
these heuristics will help novices develop expertise. The results of this study suggest that in addition to teaching 
novice instructional designers the foundational models and theories, we should also be including business and 
project management content. 

Limitations and Suggestions for Future Research 

One limitation to this study is the number of participants. Sixteen instructional designers took part in this 
study. A number of them (14) are also academicians in addition to being instructional designers. Therefore, their 
fill-time profession is not as an instructional designer. They were chosen because of their experience in the field as 
well as their willingness to be interviewed both at AECT and in the following three months. The fields in which 
these 16 instructional designers worked did not represent all possible areas of instructional design; for example, 
designers working in the military were not included. This is because the researchers did not have access to any 
instructional designers currently working for the military. In order to create a more comprehensive list of heuristics 
and get a better sense of shared heuristics the number of participants would need to be increased as well as the fields 
in which they work. 

A second limitation was the different modes of communication for interviews. Interviews in consisted of 
face-to-face, video-conference, and telephone interviews. The varying modes could have produced different results. 
The face-to-face interviews might have provided for more thoughtful responses, whereas the telephone interviews 
might reduce interviewer effects. 

Another limitation is the amount of access the researchers had to the participants. Interviews consisted of 
only one 30-90 minute interview. Follow-up interviews might verify the use of heuristics not mentioned in the initial 
interview. Another limitation is the context of the story told in the interviews. Different contexts might result in 
different heuristics (e.g., military, industry, academia). 

Future research will be devoted to validating these heuristics via the Delphi technique using a panel of 
carefully selected participants, different from the ones for this phase. The Delphi panel will consist of full-time 
practicing instructional designers. In addition, the panel will be comprised of enough participants from different 
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fields (e.g., military, industry, academia) that the researchers can examine if there is a difference in heuristics used in 
the different fields. In addition, the researchers will focus on determining the best methods by which to share the 
resulting rules of thumb with novice designers and whether it impacts their experiences as instructional designers. 
Some questions we would like to ask are: 1) Can we teach heuristics to novice instructional designers? 2) What 
methods should we use to provide this information (stories, cases, guest speakers)? And 3) How does this impact 
their practice? 

Conclusion 

“Procedures for intelligently applying past knowledge to new experience often seem to require common 
sense and practical rules of thumb in addition to, or instead of, formal analysis” (Schank & Abelson, 1977, p. 3). 

This study elicited these rules of thumb from experienced instructional designers as a way of expanding our 
understanding of what practicing instructional designers utilize during their design process. This knowledge of 
instructional design heuristics could be imparted on novice instructional designers while they are learning the 
intricacies of becoming an instructional designer possibly as a way of providing them some wisdom from more 
experienced designers. 
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Appendix 

Participants’ Demographics and Description of Story Contexts 



Participant 


Years of 

Professional 

ID 

Experience 


Current Work Context 
(years in position/rank) 


Story Context 


Bichelmeyer 


20 


Associate Professor (6 years) 
Faculty (14 years) 


Industry - large telecommunications company 
Creating a new employee orientation program and 
building a company culture 


Boling 


25 


Associate Professor (15 
years) 


Small independent company working with a large 
publishing firm to create early reading software 
(using Apple He’s) 


Branch 


24 


Professor (8 years) 
Faculty (25 years) 


Training company (intern) 

Developed tax training for accountants working with 
clients of significant net worth 
Consulted with transportation company; led 
development of training development model for the 
learning service unit. 


Cates 


34 


Associate Dean (2 years) 
Professor (34 years) 


Department of Defense (consultant) 

Building a multimedia engine for teaching thinking 
skills to middle school students 


Dabbagh 


11 


Associate Professor (3 years) 


Academia/Industry - Project manager 
Managing an internship experience for students, 
partnered with an underground coal mining 
operation, to create a refresher training system for 
supervisors 


Gibbons 


33 


Chair - Instructional 
Psychology & Technology 
Dept (4 years) 

Faculty (15 years) 


Military (consultant) 

Multiple projects described (capturing the evolution 
of his design thinking) including: 

1) Redesigning an existing course for marine 
officers (how to plan an amphibious invasion). 

2) Creating training to teach helicopter pilots 
how to conduct anti-submarine warfare 


Grant 


10 


Assistant Professor (5.5 
years) 


Academia 

Creating an assessment of web-based instruction for 
another department 


Loughner 


15 


President of Loughner and 
Associates Inc. (10 years) - 
Company that develops 
custom content training 


Pharmaceutical company; lean operations initiative 
leading to redesign of work processes and 
organizational change 


Merrill 


44 


Retired from academia 
Visiting professor FSU (2 
years) 

Consulting Prof B YU Flawaii 
(3 years) 


Center for International Entrepreneurship 
(consultant) 

Creating online business course for non-business 
majors using real-world examples 


Parrish 


20 


Managing Instmctional 
Designer (9 years) - 
Atmospheric research group 


National weather agency 

Introducing (demonstrating the benefits of) a small 
scale numerical weather prediction model 


Pedersen 


10 


Associate Professor (2 years) 
Faculty (8 years) 


NSF grant - Project Manager 
Creating virtual learning environments for middle 
school science, designed to increase student 
engagement and ability to do scientific inquiry 


Quinn 


19 


Associate Professor (11 
years) 

Faculty (15 years) 


Academia/Industry - Project manager 
Managing apprenticeship experiences for students 
who developed training for a new version of 
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company software 


Simonson 


34 


Professor (24 years) 
Faculty (33 years) 


Military (consultant) 

US Navy - Surface Warfare Threat Matrix 
Creating software to teach naval officers how to 
identify enemy ships, weapons, and aircraft 


Spector 


20 


Associate Director - 
Learning Systems Institute 
(3.5) and 

Professor (11 years) 


Air Force Research Laboratory 
Creating CBT to teach research scientists 
acquisitions technology 


Tracey 


21 


Associate Professor (1 year) 
Faculty (7 years) 


Insurance Company (consultant) 

Moving / Closing / Condensing call-centers 


Wilson 


30 


Professor (20 years) 


Academia 

Exploring strategies, such as blogging, to increase 
student engagement in an online course 
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Abstract. This study analyzed gender differences in an online course with respect to motivational 
beliefs and use of learning strategies. Sample of the study consisted of 145 participants from an 
online course which is based on synchronous and asynchronous communication methods over the 
Internet. Motivated Strategies for Learning Questionnaire was used to assess students’ motivation 
and use of learning strategies. Multivariate analysis of variance were used to analyze the data. The 
results showed that there was no statistically significant mean difference among variables with 
respect to gender. 

1. Introduction 

Gender issues in education have been recognized as an important focus for research for a long time. When 
reviewing gender related studies, the effects of this variable are inconclusive on student experience in distance 
education. Actually, numbers of studies showed that male and female students experience the online environment 
differently with respect to several ways, such as, success, motivations, perceptions, and communication behaviors 
(e.g. Chyung, 2007; Price, 2006; Rovai & Baker, 2005;), on the other hand, several results suggested that gender 
effects are insignificant (e.g. Astleitner & Steinberg, 2005; Lu et al., 2003). 

In the literature, women and men respond differently to different aspects of their learning environments while 
attending in online courses due to their dissimilar responsibilities in their life. Since female students are tiying to 
deal with multiple roles such as mother, wife, and employee. Researchers agreed that there is a need for more 
research on gender debate about differences and similarities (e.g. Price, 2006; Rovai & Baker, 2005). Therefore, in 
this paper, we aimed at discussing gender differences in an online course with respect to motivational beliefs and use 
of learning strategies. 

2. Research Hypotheses 

The following two null hypotheses were tested in this study: 

• There were no statistically significant mean differences between male and female students with respect to 
motivational beliefs in an online course. 

• There were no statistically significant mean differences between male and female students with respect to 
use of learning strategies in an online course. 

3. Subject of the study 

The study included 145 volunteer students who attended at the online course, Middle East Technical 
University (METU) in Ankara, Turkey. Originally, one hundred ninety students were registered to the course; 
however this study included the ones who were volunteers to participate in the study. The number of male 
participants (N= 101) was greater than the number of female participants (N=44), and the participants’ age ranged 
from 20 to 40 and above. The majority of the participants’ ages were between 20 and 29 (Male N=77, Female 
N=38). The majority of the participants were university graduates and undergraduate students. 

4. Instrumentation 

The Motivated Strategies for Learning Questionnaire (MSLQ), which is a self-report questionnaire developed 
by Pintrich and colleagues (1991), was used to collect relevant data. It was adapted into Turkish by Hendricks, Ekici 
and Bulut (2000). Its pilot study was conducted by administering the scale to the students enrolled in Department of 
Foreign Languages Education at METU, Turkey. The MSLQ consists of two scales: (1) motivation and (2) use of 
learning strategies. The first scale has 31 items and six sub components: Intrinsic Goal Orientation (Intr), Extrinsic 
Goal Orientation (Extr), Task Value (Tskv), Control Beliefs (Cont), Self-Efficacy for Learning and Performance 
(Slef), and Test Anxiety (Tanx). In the learning strategies scales it has 50 items and there are nine sub components: 
Rehearsal (Reh), Elaboration (Elab), Organization (Org), Critical Thinking (CrTh), Metacognitive Self-Regulation 
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(MetaStr), Time and Study Environment (TSE), Effort Regulation (EfRg), Peer Learning (PeerL), and Help Seeking 
(HelpS). The MSLQ was distributed by the researchers to the students at the end of the online course. Multivariate 
analysis of variance were used to analyze the data. 

5. Results 

Two null hypothesis of the present study were tested at the significance level of 0.05. In order to test the first 
hypothesis regarding mean differences between male and female students, MANOVA was used with gender as 
independent variable, and six motivational beliefs as dependent variables. Wilks’ X revealed that there was no an 
overall significant mean difference among the mean scores of the six variables with respect to gender (Wilks’ X = 
0.949; F(6,146)=1.254 , p>0.05). Mean and standard deviations of measures were given in Table 1. 



Table 1. Means and Standard Deviations of Six Motivational Beliefs as a Function of Gender 



Items 


# of 
items 


^^male 


SDmale 


Item 

Index 

Mean mak . 


Mf ema i e 


SDfemale 


Item 

Index 

Mean femal 

e 


M tota i 


SD t() tal 


Item 

Index 

Mean total 


Intr 


4 


22.03 


3.76 


5.51 


20.91 


4.18 


5.23 


21.68 


3.91 


5.42 


Extr 


4 


16.98 


5.06 


4.24 


17.09 


4.83 


4.27 


17.01 


4.98 


4.25 


Tskv 


6 


35.33 


4.72 


5.89 


34.89 


4.44 


5.82 


35.19 


4.63 


5.87 


Cont 


4 


22.67 


3.56 


5.67 


22.78 


3.55 


5.70 


22.71 


3.54 


5.68 


Slef 


8 


41.45 


7.48 


5.18 


38.84 


7.63 


4.86 


40.65 


7.60 


5.08 


Tanx 


5 


18.14 


6.36 


3.63 


19.52 


6.32 


3.90 


18.57 


6.36 


3.71 



As seen in Table 1, it could be stated that male and female learners had the same tendency in motivational 
beliefs. Furthermore, they tended to reflect an “undecided” perspective toward their motivational beliefs regarding 
extrinsic goal orientation and test anxiety. Also, they tended to reflect an “agree” perspective toward their 
motivational beliefs with regard to the other dimensions of motivational scales. But these beliefs were not so strong. 

In order to test the second hypothesis regarding mean difference between male and female students, 
MANOVA was again used with gender as independent variable, and nine variables related to use of learning 
strategies as dependent variables. Wilks’ X revealed that there was no an overall significant mean difference among 
the mean scores of the nine variables with respect to gender (Wilks’ X = 0.920; F(9,146)= 1.3 10, p>0.05). Mean and 
standard deviations of measures were given in Table 2. 



Table 2. Means and Standard Deviations of Nine Learning Strategies as a Function of Gender 



Items 


# of 
items 


M ma i e 


SDmale 


Item 

Index 

Mean ma ie 


^Ifemale 


SDfemale 


Item 

Index 

Mean fema i e 


M^tal 


SD total 


Item 

Index 

Mean tota i 


Reh 


4 


14.87 


4.95 


3.72 


15.33 


5.17 


3.83 


15.01 


5.00 


3.75 


Elab 


6 


29.66 


6.09 


4.94 


30.96 


4.57 


5.16 


30.06 


5.68 


5.01 


Org 


4 


19.18 


4.49 


4.80 


20.76 


3.16 


5.19 


19.67 


4.18 


4.92 


CrTh 


5 


23.41 


4.66 


4.68 


22.90 


5.53 


4.58 


23.26 


4.93 


4.65 


MetaStr 


12 


57.24 


10.42 


4.77 


60.47 


9.81 


5.04 


58.23 


10.31 


4.85 


TSE 


8 


34.40 


6.05 


4.30 


33.76 


7.66 


4.22 


34.20 


6.56 


4.28 


EfRg 


4 


17.31 


4.29 


4.33 


17.69 


4.33 


4.42 


17.42 


4.29 


4.36 


PeerL 


3 


9.44 


4.10 


3.15 


10.02 


3.35 


3.34 


9.62 


3.88 


3.21 


HelpS 


4 


15.12 


5.07 


3.78 


16.58 


4.68 


4.14 


15.57 


4.99 


3.89 



As seen in Table 2, it could be stated that male and female learners had the same tendency in the use of 
learning strategies. Moreover, they tended to reflect an “undecided” perspective toward their use of learning 
strategies regarding time and study environment and effort regulation. Also, it could be stated that they tended to 
reflect a “disagree” perspective toward their use of learning strategies regarding peer learning and help seeking. 
They also accepted the items in other dimensions but their beliefs were not very strong again. 
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6. Conclusion 

This study analyzed gender differences in an online course with respect to motivational beliefs and use of 
learning strategies. The results showed that there was no statistically significant mean difference among variables 
with respect to gender. These findings were supported by many of the existing findings from the literature. In other 
words, although several researchers stated that female and male students experience the online environment 
differently, gender in several learning environment were not reported as significant variable in the past (e.g. 
Astleitner & Steinberg, 2005; Lu et al., 2003). It can be stated that the gender-related studies might show similar 
results in different society. Moreover, the lack of gender-related differences might be partly explained by the 
participants’ properties and nature of the online course. Online learners are mature enough and they are aware of 
their responsibilities and the course was offered with help of synchronous and asynchronous communication 
methods over the Internet in three months. 
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